2026-02-24T16:30:48.154 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T16:30:48.170 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:30:48.251 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330 branch: wip-yuri8-testing-2026-02-23-2128 description: nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_namespaces} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '68330' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2026-02-23-2128 ceph: conf: global: mon warn on pool no app: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - OSD_DOWN - OSD_HOST_DOWN - OSD_ROOT_DOWN - PG_DEGRADED - NVMEOF_SINGLE_GATEWAY - CEPHADM_FAILED_DAEMON sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 workunit: branch: wip-yuri8-testing-2026-02-23-2128 sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - client.0 - - client.1 - - host.a - mon.a - mgr.x - osd.0 - client.2 - ceph.nvmeof.nvmeof.a - - host.b - mon.b - osd.1 - client.3 - ceph.nvmeof.nvmeof.b - - host.c - mon.c - osd.2 - client.4 - ceph.nvmeof.nvmeof.c - - host.d - osd.3 - client.5 - ceph.nvmeof.nvmeof.d seed: 4497 sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 sleep_before_teardown: 0 subset: 111/120000 suite: nvmeof suite_branch: wip-yuri8-testing-2026-02-23-2128 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 targets: trial051.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEMfdyr2ASV1FQTwn/nU/4Q/qU2eqYTLuFcs6x4058EXUnpUDE1QeDom9NdyxVKk0fVLEswpnHWAzi5UOhxzcns= trial052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKYGZMlC6GyPnLrmQeB+ta4No/OJqYSjUttyhfqnoqYop6u5OF06yz0xg5TqB0sBXhWCcxlgsEeXuJWaEXENkiE= trial151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB7wQL7ozvrKP22dnXbKLa033UHfrrlaT2Ct77/vzgpgoiFcjHCTavrLsXXlJxaZqQb1M1Nnby9INgUL94L0G8= trial153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBII8Jags94HTt3o8/RIVs009gxPw7qKN/N4tUDb3/uYU603/a/cBPKX/9TbRSRbnAVagICxMeMaTfMRqH2+gz+o= trial183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMJ3khXTl/KQ29xFw30kWB2ySfPJ0gJbpXg6E571xlUCAffqbl04D4vJFtjTBKAXu4ZiPyAvx5qB17pMwVuoddw= trial191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMQUHsDh/pUjsnXJw/jIEc0EIZ8BDSJRQB1PrlqB2BuDFPUT+0gPyzn9Ygziu/YT69FkZD6gtMvkk9ZalNhbb5Y= tasks: - install: extra_packages: - nvme-cli - cephadm: null - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch host ls - ceph orch device ls - ceph osd lspools - nvmeof: gateway_config: cli_image: quay.io/ceph/nvmeof-cli:devel namespaces_count: 20 subsystems_count: 3 gw_image: quay.io/ceph/nvmeof:devel installer: host.a rbd: image_name_prefix: myimage pool_name: mypool - cephadm.wait_for_service: refresh: true service: nvmeof.mypool.mygroup0 - workunit: clients: client.0: - nvmeof/setup_subsystem.sh - nvmeof/basic_tests.sh env: RBD_IMAGE_PREFIX: myimage RBD_POOL: mypool no_coverage_and_limits: true - workunit: clients: client.0: - nvmeof/fio_test.sh --rbd_iostat client.1: - nvmeof/basic_tests.sh - nvmeof/namespace_test.sh env: IOSTAT_INTERVAL: '10' NEW_NAMESPACES_COUNT: '5' RBD_POOL: mypool RUNTIME: '120' no_coverage_and_limits: true timeout: 30m teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_16:03:54 tube: trial use_shaman: true user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T16:30:48.252 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa; will attempt to use it 2026-02-24T16:30:48.252 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa/tasks 2026-02-24T16:30:48.252 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T16:30:48.254 INFO:teuthology.task.internal:Checking packages... 2026-02-24T16:30:48.254 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '618c318a7a0b287ab5aeff207b7bf1d73ef45044' 2026-02-24T16:30:48.254 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T16:30:48.254 INFO:teuthology.packaging:ref: None 2026-02-24T16:30:48.254 INFO:teuthology.packaging:tag: None 2026-02-24T16:30:48.254 INFO:teuthology.packaging:branch: wip-yuri8-testing-2026-02-23-2128 2026-02-24T16:30:48.254 INFO:teuthology.packaging:sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:30:48.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2026-02-23-2128 2026-02-24T16:30:48.381 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5507.g618c318a 2026-02-24T16:30:48.383 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T16:30:48.396 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T16:30:48.396 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T16:30:48.410 INFO:teuthology.task.internal:Saving configuration 2026-02-24T16:30:48.424 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T16:30:48.437 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T16:30:48.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial051.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.668668', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEMfdyr2ASV1FQTwn/nU/4Q/qU2eqYTLuFcs6x4058EXUnpUDE1QeDom9NdyxVKk0fVLEswpnHWAzi5UOhxzcns='} 2026-02-24T16:30:48.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial052.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.667240', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKYGZMlC6GyPnLrmQeB+ta4No/OJqYSjUttyhfqnoqYop6u5OF06yz0xg5TqB0sBXhWCcxlgsEeXuJWaEXENkiE='} 2026-02-24T16:30:48.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial151.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.670135', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB7wQL7ozvrKP22dnXbKLa033UHfrrlaT2Ct77/vzgpgoiFcjHCTavrLsXXlJxaZqQb1M1Nnby9INgUL94L0G8='} 2026-02-24T16:30:48.818 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial153.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.667852', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBII8Jags94HTt3o8/RIVs009gxPw7qKN/N4tUDb3/uYU603/a/cBPKX/9TbRSRbnAVagICxMeMaTfMRqH2+gz+o='} 2026-02-24T16:30:48.874 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial183.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.668274', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMJ3khXTl/KQ29xFw30kWB2ySfPJ0gJbpXg6E571xlUCAffqbl04D4vJFtjTBKAXu4ZiPyAvx5qB17pMwVuoddw='} 2026-02-24T16:30:48.929 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial191.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 16:20:36.669752', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMQUHsDh/pUjsnXJw/jIEc0EIZ8BDSJRQB1PrlqB2BuDFPUT+0gPyzn9Ygziu/YT69FkZD6gtMvkk9ZalNhbb5Y='} 2026-02-24T16:30:48.929 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial051.front.sepia.ceph.com - ['client.0'] 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial052.front.sepia.ceph.com - ['client.1'] 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial151.front.sepia.ceph.com - ['host.a', 'mon.a', 'mgr.x', 'osd.0', 'client.2', 'ceph.nvmeof.nvmeof.a'] 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial153.front.sepia.ceph.com - ['host.b', 'mon.b', 'osd.1', 'client.3', 'ceph.nvmeof.nvmeof.b'] 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial183.front.sepia.ceph.com - ['host.c', 'mon.c', 'osd.2', 'client.4', 'ceph.nvmeof.nvmeof.c'] 2026-02-24T16:30:48.940 INFO:teuthology.task.internal:roles: ubuntu@trial191.front.sepia.ceph.com - ['host.d', 'osd.3', 'client.5', 'ceph.nvmeof.nvmeof.d'] 2026-02-24T16:30:48.940 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T16:30:49.300 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e486ca12ca0>, signals=[15]) 2026-02-24T16:30:49.300 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T16:30:49.311 INFO:teuthology.task.internal:Opening connections... 2026-02-24T16:30:49.311 DEBUG:teuthology.task.internal:connecting to ubuntu@trial051.front.sepia.ceph.com 2026-02-24T16:30:49.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.377 DEBUG:teuthology.task.internal:connecting to ubuntu@trial052.front.sepia.ceph.com 2026-02-24T16:30:49.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.440 DEBUG:teuthology.task.internal:connecting to ubuntu@trial151.front.sepia.ceph.com 2026-02-24T16:30:49.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.502 DEBUG:teuthology.task.internal:connecting to ubuntu@trial153.front.sepia.ceph.com 2026-02-24T16:30:49.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.565 DEBUG:teuthology.task.internal:connecting to ubuntu@trial183.front.sepia.ceph.com 2026-02-24T16:30:49.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.628 DEBUG:teuthology.task.internal:connecting to ubuntu@trial191.front.sepia.ceph.com 2026-02-24T16:30:49.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:30:49.690 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T16:30:49.701 DEBUG:teuthology.orchestra.run.trial051:> uname -m 2026-02-24T16:30:49.718 INFO:teuthology.orchestra.run.trial051.stdout:x86_64 2026-02-24T16:30:49.719 DEBUG:teuthology.orchestra.run.trial051:> cat /etc/os-release 2026-02-24T16:30:49.776 INFO:teuthology.orchestra.run.trial051.stdout:NAME="CentOS Stream" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:VERSION="9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:ID="centos" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:VERSION_ID="9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:49.777 INFO:teuthology.orchestra.run.trial051.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:49.777 INFO:teuthology.lock.ops:Updating trial051.front.sepia.ceph.com on lock server 2026-02-24T16:30:49.845 DEBUG:teuthology.orchestra.run.trial052:> uname -m 2026-02-24T16:30:49.862 INFO:teuthology.orchestra.run.trial052.stdout:x86_64 2026-02-24T16:30:49.862 DEBUG:teuthology.orchestra.run.trial052:> cat /etc/os-release 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:NAME="CentOS Stream" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:VERSION="9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:ID="centos" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:VERSION_ID="9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:49.918 INFO:teuthology.orchestra.run.trial052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:49.919 INFO:teuthology.lock.ops:Updating trial052.front.sepia.ceph.com on lock server 2026-02-24T16:30:49.993 DEBUG:teuthology.orchestra.run.trial151:> uname -m 2026-02-24T16:30:50.011 INFO:teuthology.orchestra.run.trial151.stdout:x86_64 2026-02-24T16:30:50.011 DEBUG:teuthology.orchestra.run.trial151:> cat /etc/os-release 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:NAME="CentOS Stream" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:VERSION="9" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:ID="centos" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:VERSION_ID="9" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:50.066 INFO:teuthology.orchestra.run.trial151.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:50.067 INFO:teuthology.orchestra.run.trial151.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:50.067 INFO:teuthology.orchestra.run.trial151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:50.067 INFO:teuthology.orchestra.run.trial151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:50.067 INFO:teuthology.lock.ops:Updating trial151.front.sepia.ceph.com on lock server 2026-02-24T16:30:50.140 DEBUG:teuthology.orchestra.run.trial153:> uname -m 2026-02-24T16:30:50.157 INFO:teuthology.orchestra.run.trial153.stdout:x86_64 2026-02-24T16:30:50.157 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:50.212 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:50.213 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:50.213 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server 2026-02-24T16:30:50.295 DEBUG:teuthology.orchestra.run.trial183:> uname -m 2026-02-24T16:30:50.310 INFO:teuthology.orchestra.run.trial183.stdout:x86_64 2026-02-24T16:30:50.310 DEBUG:teuthology.orchestra.run.trial183:> cat /etc/os-release 2026-02-24T16:30:50.366 INFO:teuthology.orchestra.run.trial183.stdout:NAME="CentOS Stream" 2026-02-24T16:30:50.366 INFO:teuthology.orchestra.run.trial183.stdout:VERSION="9" 2026-02-24T16:30:50.366 INFO:teuthology.orchestra.run.trial183.stdout:ID="centos" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:VERSION_ID="9" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:50.367 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:50.367 INFO:teuthology.lock.ops:Updating trial183.front.sepia.ceph.com on lock server 2026-02-24T16:30:50.449 DEBUG:teuthology.orchestra.run.trial191:> uname -m 2026-02-24T16:30:50.466 INFO:teuthology.orchestra.run.trial191.stdout:x86_64 2026-02-24T16:30:50.467 DEBUG:teuthology.orchestra.run.trial191:> cat /etc/os-release 2026-02-24T16:30:50.523 INFO:teuthology.orchestra.run.trial191.stdout:NAME="CentOS Stream" 2026-02-24T16:30:50.523 INFO:teuthology.orchestra.run.trial191.stdout:VERSION="9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:ID="centos" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:ID_LIKE="rhel fedora" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:VERSION_ID="9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:PLATFORM_ID="platform:el9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:ANSI_COLOR="0;31" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:LOGO="fedora-logo-icon" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:HOME_URL="https://centos.org/" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T16:30:50.524 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T16:30:50.525 INFO:teuthology.lock.ops:Updating trial191.front.sepia.ceph.com on lock server 2026-02-24T16:30:50.607 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T16:30:50.620 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T16:30:50.631 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T16:30:50.631 DEBUG:teuthology.orchestra.run.trial051:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.633 DEBUG:teuthology.orchestra.run.trial052:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.635 DEBUG:teuthology.orchestra.run.trial151:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.637 DEBUG:teuthology.orchestra.run.trial153:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.639 DEBUG:teuthology.orchestra.run.trial183:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.641 DEBUG:teuthology.orchestra.run.trial191:> test '!' -e /home/ubuntu/cephtest 2026-02-24T16:30:50.655 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T16:30:50.667 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T16:30:50.667 DEBUG:teuthology.orchestra.run.trial051:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.688 DEBUG:teuthology.orchestra.run.trial052:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.690 DEBUG:teuthology.orchestra.run.trial151:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.694 DEBUG:teuthology.orchestra.run.trial153:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.695 DEBUG:teuthology.orchestra.run.trial183:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.697 DEBUG:teuthology.orchestra.run.trial191:> test -z $(ls -A /var/lib/ceph) 2026-02-24T16:30:50.714 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T16:30:51.093 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T16:30:51.115 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T16:30:51.115 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'client.1': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'host.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'host.b': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'host.c': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'host.d': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T16:30:51.115 DEBUG:teuthology.orchestra.run.trial051:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.115 DEBUG:teuthology.orchestra.run.trial052:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.115 DEBUG:teuthology.orchestra.run.trial151:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.116 DEBUG:teuthology.orchestra.run.trial153:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.116 DEBUG:teuthology.orchestra.run.trial183:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.116 DEBUG:teuthology.orchestra.run.trial191:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run.trial052:> uname -r 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run.trial191:> uname -r 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run.trial153:> uname -r 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run.trial051:> uname -r 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run.trial151:> uname -r 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:30:51.133 DEBUG:teuthology.orchestra.run.trial183:> uname -r 2026-02-24T16:30:51.187 INFO:teuthology.orchestra.run.trial052.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.188 INFO:teuthology.task.kernel:Running kernel on trial052: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.188 DEBUG:teuthology.orchestra.run.trial052:> sudo yum install -y kernel 2026-02-24T16:30:51.188 INFO:teuthology.orchestra.run.trial153.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.188 INFO:teuthology.task.kernel:Running kernel on trial153: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.188 DEBUG:teuthology.orchestra.run.trial153:> sudo yum install -y kernel 2026-02-24T16:30:51.189 INFO:teuthology.orchestra.run.trial051.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.189 INFO:teuthology.orchestra.run.trial151.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.189 INFO:teuthology.task.kernel:Running kernel on trial051: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.189 DEBUG:teuthology.orchestra.run.trial051:> sudo yum install -y kernel 2026-02-24T16:30:51.189 INFO:teuthology.task.kernel:Running kernel on trial151: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.189 DEBUG:teuthology.orchestra.run.trial151:> sudo yum install -y kernel 2026-02-24T16:30:51.190 INFO:teuthology.orchestra.run.trial183.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.190 INFO:teuthology.orchestra.run.trial191.stdout:5.14.0-681.el9.x86_64 2026-02-24T16:30:51.190 INFO:teuthology.task.kernel:Running kernel on trial183: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.190 DEBUG:teuthology.orchestra.run.trial183:> sudo yum install -y kernel 2026-02-24T16:30:51.190 INFO:teuthology.task.kernel:Running kernel on trial191: 5.14.0-681.el9.x86_64 2026-02-24T16:30:51.190 DEBUG:teuthology.orchestra.run.trial191:> sudo yum install -y kernel 2026-02-24T16:30:51.902 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - BaseOS 27 MB/s | 8.9 MB 00:00 2026-02-24T16:30:52.346 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-24T16:30:52.442 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-24T16:30:53.379 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.9 MB 00:01 2026-02-24T16:30:53.473 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.9 MB 00:01 2026-02-24T16:30:53.764 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.9 MB 00:02 2026-02-24T16:30:55.046 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-24T16:30:55.185 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - AppStream 12 MB/s | 27 MB 00:02 2026-02-24T16:30:55.601 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-24T16:30:56.004 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-24T16:30:57.167 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 27 MB 00:03 2026-02-24T16:30:57.361 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 27 MB 00:03 2026-02-24T16:30:58.483 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 8.0 MB 00:01 2026-02-24T16:30:59.271 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 8.0 MB 00:01 2026-02-24T16:30:59.353 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 8.0 MB 00:00 2026-02-24T16:30:59.381 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - Extras packages 124 kB/s | 20 kB 00:00 2026-02-24T16:31:00.093 INFO:teuthology.orchestra.run.trial191.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-24T16:31:00.297 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 20 kB 00:00 2026-02-24T16:31:00.424 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 8.0 MB 00:00 2026-02-24T16:31:00.468 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-24T16:31:00.676 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 8.0 MB 00:00 2026-02-24T16:31:00.747 INFO:teuthology.orchestra.run.trial051.stdout:Extra Packages for Enterprise Linux 95 MB/s | 20 MB 00:00 2026-02-24T16:31:01.925 INFO:teuthology.orchestra.run.trial151.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-24T16:31:02.030 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-24T16:31:02.529 INFO:teuthology.orchestra.run.trial052.stdout:Extra Packages for Enterprise Linux 47 MB/s | 20 MB 00:00 2026-02-24T16:31:02.532 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-24T16:31:03.288 INFO:teuthology.orchestra.run.trial191.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-24T16:31:03.780 INFO:teuthology.orchestra.run.trial153.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-24T16:31:03.914 INFO:teuthology.orchestra.run.trial051.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T16:31:04.313 INFO:teuthology.orchestra.run.trial191.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:04.313 INFO:teuthology.orchestra.run.trial191.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:04.333 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:31:04.334 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:31:04.334 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:04.335 INFO:teuthology.orchestra.run.trial191.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:Install 4 Packages 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 89 M 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 135 M 2026-02-24T16:31:04.336 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:31:04.941 INFO:teuthology.orchestra.run.trial051.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:04.941 INFO:teuthology.orchestra.run.trial051.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:04.961 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:31:04.962 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:31:04.962 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:04.962 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:31:04.962 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:31:04.962 INFO:teuthology.orchestra.run.trial051.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Install 4 Packages 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 89 M 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 135 M 2026-02-24T16:31:04.963 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:31:05.126 INFO:teuthology.orchestra.run.trial151.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-24T16:31:05.553 INFO:teuthology.orchestra.run.trial191.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 990 kB/s | 901 kB 00:00 2026-02-24T16:31:05.733 INFO:teuthology.orchestra.run.trial052.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T16:31:06.023 INFO:teuthology.orchestra.run.trial191.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2026-02-24T16:31:06.151 INFO:teuthology.orchestra.run.trial151.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:06.151 INFO:teuthology.orchestra.run.trial151.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:06.171 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:31:06.172 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:31:06.172 INFO:teuthology.orchestra.run.trial151.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:06.172 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:31:06.172 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:31:06.172 INFO:teuthology.orchestra.run.trial151.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Installing dependencies: 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Install 4 Packages 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 89 M 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 135 M 2026-02-24T16:31:06.173 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:31:06.359 INFO:teuthology.orchestra.run.trial151.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 6.9 MB/s | 901 kB 00:00 2026-02-24T16:31:06.516 INFO:teuthology.orchestra.run.trial191.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 21 MB/s | 40 MB 00:01 2026-02-24T16:31:06.569 INFO:teuthology.orchestra.run.trial151.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 51 MB/s | 17 MB 00:00 2026-02-24T16:31:06.683 INFO:teuthology.orchestra.run.trial051.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 543 kB/s | 901 kB 00:01 2026-02-24T16:31:06.745 INFO:teuthology.orchestra.run.trial052.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:06.745 INFO:teuthology.orchestra.run.trial052.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:06.763 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 927 kB/s | 8.0 MB 00:08 2026-02-24T16:31:06.765 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:31:06.766 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:31:06.766 INFO:teuthology.orchestra.run.trial052.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:06.766 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:31:06.766 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:31:06.766 INFO:teuthology.orchestra.run.trial052.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Installing dependencies: 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Install 4 Packages 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 89 M 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 135 M 2026-02-24T16:31:06.767 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:31:06.777 INFO:teuthology.orchestra.run.trial151.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 74 MB/s | 31 MB 00:00 2026-02-24T16:31:06.874 INFO:teuthology.orchestra.run.trial151.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 62 MB/s | 40 MB 00:00 2026-02-24T16:31:06.874 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:06.874 INFO:teuthology.orchestra.run.trial151.stdout:Total 127 MB/s | 89 MB 00:00 2026-02-24T16:31:06.920 INFO:teuthology.orchestra.run.trial153.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T16:31:07.067 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:31:07.277 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:31:07.277 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:31:07.280 INFO:teuthology.orchestra.run.trial191.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 18 MB/s | 31 MB 00:01 2026-02-24T16:31:07.280 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:07.281 INFO:teuthology.orchestra.run.trial191.stdout:Total 30 MB/s | 89 MB 00:02 2026-02-24T16:31:07.473 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:31:07.662 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:31:07.662 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:31:07.681 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:31:07.681 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:31:07.932 INFO:teuthology.orchestra.run.trial153.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:07.932 INFO:teuthology.orchestra.run.trial153.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:07.952 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:31:07.953 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:31:07.953 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:07.953 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout:Install 4 Packages 2026-02-24T16:31:07.954 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:07.955 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 89 M 2026-02-24T16:31:07.955 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 135 M 2026-02-24T16:31:07.955 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:31:07.982 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 20 kB 00:00 2026-02-24T16:31:07.985 INFO:teuthology.orchestra.run.trial052.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 985 kB/s | 901 kB 00:00 2026-02-24T16:31:08.068 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:31:08.068 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:31:08.171 INFO:teuthology.orchestra.run.trial153.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 6.9 MB/s | 901 kB 00:00 2026-02-24T16:31:08.381 INFO:teuthology.orchestra.run.trial153.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 51 MB/s | 17 MB 00:00 2026-02-24T16:31:08.436 INFO:teuthology.orchestra.run.trial052.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 13 MB/s | 17 MB 00:01 2026-02-24T16:31:08.529 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 42 MB/s | 20 MB 00:00 2026-02-24T16:31:08.592 INFO:teuthology.orchestra.run.trial153.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 74 MB/s | 31 MB 00:00 2026-02-24T16:31:08.711 INFO:teuthology.orchestra.run.trial153.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 60 MB/s | 40 MB 00:00 2026-02-24T16:31:08.711 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:08.711 INFO:teuthology.orchestra.run.trial153.stdout:Total 117 MB/s | 89 MB 00:00 2026-02-24T16:31:08.764 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:31:08.906 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:31:08.961 INFO:teuthology.orchestra.run.trial052.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 21 MB/s | 40 MB 00:01 2026-02-24T16:31:09.053 INFO:teuthology.orchestra.run.trial151.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:09.075 INFO:teuthology.orchestra.run.trial151.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:09.107 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:31:09.108 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:31:09.181 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:31:09.469 INFO:teuthology.orchestra.run.trial191.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:09.491 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:31:09.491 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:31:09.492 INFO:teuthology.orchestra.run.trial191.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:09.679 INFO:teuthology.orchestra.run.trial052.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 18 MB/s | 31 MB 00:01 2026-02-24T16:31:09.680 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:09.680 INFO:teuthology.orchestra.run.trial052.stdout:Total 31 MB/s | 89 MB 00:02 2026-02-24T16:31:09.814 INFO:teuthology.orchestra.run.trial051.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 3.6 MB/s | 17 MB 00:04 2026-02-24T16:31:09.871 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:31:10.078 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:31:10.079 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:31:10.460 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:31:10.460 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:31:10.602 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:31:10.891 INFO:teuthology.orchestra.run.trial153.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:10.912 INFO:teuthology.orchestra.run.trial153.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:11.569 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:31:11.638 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:11.670 INFO:teuthology.orchestra.run.trial151.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:11.704 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-24T16:31:11.859 INFO:teuthology.orchestra.run.trial052.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:11.881 INFO:teuthology.orchestra.run.trial052.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:12.073 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:12.106 INFO:teuthology.orchestra.run.trial191.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:12.718 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T16:31:12.718 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T16:31:12.739 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T16:31:12.740 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:Install 4 Packages 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 89 M 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 135 M 2026-02-24T16:31:12.741 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:31:12.862 INFO:teuthology.orchestra.run.trial183.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 13 MB/s | 901 kB 00:00 2026-02-24T16:31:13.019 INFO:teuthology.orchestra.run.trial183.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 77 MB/s | 17 MB 00:00 2026-02-24T16:31:13.346 INFO:teuthology.orchestra.run.trial183.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 73 MB/s | 40 MB 00:00 2026-02-24T16:31:13.466 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:13.499 INFO:teuthology.orchestra.run.trial153.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:13.746 INFO:teuthology.orchestra.run.trial183.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 35 MB/s | 31 MB 00:00 2026-02-24T16:31:13.747 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:13.747 INFO:teuthology.orchestra.run.trial183.stdout:Total 88 MB/s | 89 MB 00:01 2026-02-24T16:31:13.939 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:31:14.143 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:31:14.144 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:31:14.435 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:14.467 INFO:teuthology.orchestra.run.trial052.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:14.531 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:31:14.531 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:31:14.894 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:14.902 INFO:teuthology.orchestra.run.trial151.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:15.147 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:15.153 INFO:teuthology.orchestra.run.trial191.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:15.646 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:31:15.938 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:15.960 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:16.534 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:16.541 INFO:teuthology.orchestra.run.trial153.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:17.509 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:17.515 INFO:teuthology.orchestra.run.trial052.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:17.954 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:18.401 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:18.522 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:18.555 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:19.650 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:20.589 INFO:teuthology.orchestra.run.trial051.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 2.6 MB/s | 40 MB 00:15 2026-02-24T16:31:20.629 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:21.595 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:21.602 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:24.655 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.502 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.524 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.714 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.714 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:35.715 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:35.715 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:35.772 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:31:35.839 DEBUG:teuthology.orchestra.run.trial151:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:35.869 INFO:teuthology.orchestra.run.trial151.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.869 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:35.870 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:31:35.870 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:31:35.870 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:31:35.870 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2026-02-24T16:31:35.870 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial151.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:31:35.870 DEBUG:teuthology.orchestra.run.trial151:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:35.939 INFO:teuthology.orchestra.run.trial151.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:35.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:35.941 DEBUG:teuthology.orchestra.run.trial151:> sudo rpm -qi grub2-tools 2026-02-24T16:31:35.968 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:35.990 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Name : grub2-tools 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Epoch : 1 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Version : 2.06 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Release : 120.el9 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Architecture: x86_64 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:31:36.011 INFO:teuthology.orchestra.run.trial151.stdout:Group : Unspecified 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Size : 9108611 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:License : GPLv3+ 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Packager : builder@centos.org 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Vendor : CentOS 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Summary : Support tools for GRUB. 2026-02-24T16:31:36.012 INFO:teuthology.orchestra.run.trial151.stdout:Description : 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout:hardware devices. 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:31:36.013 INFO:teuthology.orchestra.run.trial151.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:31:36.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:31:36.014 INFO:teuthology.task.kernel:Updating grub on trial151 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:31:36.015 DEBUG:teuthology.orchestra.run.trial151:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:31:36.079 INFO:teuthology.orchestra.run.trial151.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:31:36.081 DEBUG:teuthology.orchestra.run.trial151:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:31:36.143 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/grubby 2026-02-24T16:31:36.145 INFO:teuthology.orchestra.run.trial151.stdout:yes 2026-02-24T16:31:36.146 DEBUG:teuthology.orchestra.run.trial151:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:31:36.177 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:36.177 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:36.177 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:36.177 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:36.209 INFO:teuthology.orchestra.run.trial151.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:31:36.210 DEBUG:teuthology.orchestra.run.trial151:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.234 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:36.234 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:36.234 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:36.235 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:31:36.283 DEBUG:teuthology.orchestra.run.trial151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:31:36.296 DEBUG:teuthology.orchestra.run.trial191:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:36.325 INFO:teuthology.orchestra.run.trial191.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.327 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:36.327 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:31:36.328 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:31:36.328 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:31:36.328 INFO:teuthology.task.kernel:Installing distro kernel on host.d... 2026-02-24T16:31:36.328 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial191.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:31:36.328 DEBUG:teuthology.orchestra.run.trial191:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:36.394 INFO:teuthology.orchestra.run.trial191.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.396 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:36.396 DEBUG:teuthology.orchestra.run.trial191:> sudo rpm -qi grub2-tools 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Name : grub2-tools 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Epoch : 1 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Version : 2.06 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Release : 120.el9 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Architecture: x86_64 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:31:36.465 INFO:teuthology.orchestra.run.trial191.stdout:Group : Unspecified 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Size : 9108611 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:License : GPLv3+ 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Packager : builder@centos.org 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Vendor : CentOS 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Summary : Support tools for GRUB. 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:Description : 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:hardware devices. 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:31:36.466 INFO:teuthology.orchestra.run.trial191.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:31:36.467 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:31:36.467 INFO:teuthology.task.kernel:Updating grub on trial191 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:31:36.467 DEBUG:teuthology.orchestra.run.trial191:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:31:36.494 INFO:teuthology.orchestra.run.trial151.stderr:Generating grub configuration file ... 2026-02-24T16:31:36.529 INFO:teuthology.orchestra.run.trial191.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:31:36.530 DEBUG:teuthology.orchestra.run.trial191:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:31:36.563 INFO:teuthology.orchestra.run.trial151.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.569 INFO:teuthology.orchestra.run.trial151.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:31:36.593 INFO:teuthology.orchestra.run.trial191.stdout:/sbin/grubby 2026-02-24T16:31:36.594 INFO:teuthology.orchestra.run.trial191.stdout:yes 2026-02-24T16:31:36.595 DEBUG:teuthology.orchestra.run.trial191:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:31:36.660 INFO:teuthology.orchestra.run.trial191.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:31:36.660 DEBUG:teuthology.orchestra.run.trial191:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:31:36.735 DEBUG:teuthology.orchestra.run.trial191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:31:36.825 INFO:teuthology.orchestra.run.trial151.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:31:36.832 INFO:teuthology.orchestra.run.trial151.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:31:36.867 INFO:teuthology.orchestra.run.trial151.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:31:36.874 INFO:teuthology.orchestra.run.trial151.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:31:36.904 INFO:teuthology.orchestra.run.trial151.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:31:36.911 INFO:teuthology.orchestra.run.trial151.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:31:36.934 INFO:teuthology.orchestra.run.trial151.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:31:36.941 INFO:teuthology.orchestra.run.trial151.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:31:36.943 INFO:teuthology.orchestra.run.trial191.stderr:Generating grub configuration file ... 2026-02-24T16:31:37.011 INFO:teuthology.orchestra.run.trial191.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.018 INFO:teuthology.orchestra.run.trial191.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:31:37.028 INFO:teuthology.orchestra.run.trial151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:31:37.037 INFO:teuthology.orchestra.run.trial151.stderr:done 2026-02-24T16:31:37.039 DEBUG:teuthology.orchestra.run.trial151:> test -d /sys/firmware/efi 2026-02-24T16:31:37.056 DEBUG:teuthology.orchestra.run.trial151:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:31:37.121 INFO:teuthology.orchestra.run.trial151.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:31:37.122 INFO:teuthology.orchestra.run.trial151.stdout:/boot/efi/EFI/centos 2026-02-24T16:31:37.124 DEBUG:teuthology.orchestra.run.trial151:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:31:37.191 INFO:teuthology.orchestra.run.trial151.stdout:no 2026-02-24T16:31:37.192 DEBUG:teuthology.orchestra.run.trial151:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:31:37.256 INFO:teuthology.orchestra.run.trial151.stdout:yes 2026-02-24T16:31:37.256 DEBUG:teuthology.orchestra.run.trial151:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:31:37.271 INFO:teuthology.orchestra.run.trial191.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:31:37.278 INFO:teuthology.orchestra.run.trial191.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:31:37.312 INFO:teuthology.orchestra.run.trial191.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:31:37.318 INFO:teuthology.orchestra.run.trial191.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:31:37.320 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial151 2026-02-24T16:31:37.321 DEBUG:teuthology.orchestra.run.trial151:> sudo bash -c ' 2026-02-24T16:31:37.321 DEBUG:teuthology.orchestra.run.trial151:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:31:37.321 DEBUG:teuthology.orchestra.run.trial151:> ' 2026-02-24T16:31:37.350 INFO:teuthology.orchestra.run.trial191.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:31:37.356 INFO:teuthology.orchestra.run.trial191.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:31:37.364 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:37.380 INFO:teuthology.orchestra.run.trial191.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:31:37.387 INFO:teuthology.orchestra.run.trial191.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:31:37.388 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:37.475 INFO:teuthology.orchestra.run.trial191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:31:37.483 INFO:teuthology.orchestra.run.trial191.stderr:done 2026-02-24T16:31:37.485 DEBUG:teuthology.orchestra.run.trial191:> test -d /sys/firmware/efi 2026-02-24T16:31:37.500 DEBUG:teuthology.orchestra.run.trial191:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:31:37.567 INFO:teuthology.orchestra.run.trial191.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:31:37.567 INFO:teuthology.orchestra.run.trial191.stdout:/boot/efi/EFI/centos 2026-02-24T16:31:37.569 DEBUG:teuthology.orchestra.run.trial191:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:31:37.577 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:37.577 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:37.577 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:37.577 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:37.634 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:37.634 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:37.635 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:31:37.636 INFO:teuthology.orchestra.run.trial191.stdout:no 2026-02-24T16:31:37.637 DEBUG:teuthology.orchestra.run.trial191:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:31:37.697 DEBUG:teuthology.orchestra.run.trial153:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:37.701 INFO:teuthology.orchestra.run.trial191.stdout:yes 2026-02-24T16:31:37.702 DEBUG:teuthology.orchestra.run.trial191:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:31:37.725 INFO:teuthology.orchestra.run.trial153.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.727 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:37.727 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:31:37.727 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:31:37.727 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:31:37.727 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2026-02-24T16:31:37.727 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial153.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:31:37.727 DEBUG:teuthology.orchestra.run.trial153:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:37.767 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial191 2026-02-24T16:31:37.768 DEBUG:teuthology.orchestra.run.trial191:> sudo bash -c ' 2026-02-24T16:31:37.768 DEBUG:teuthology.orchestra.run.trial191:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:31:37.768 DEBUG:teuthology.orchestra.run.trial191:> ' 2026-02-24T16:31:37.793 INFO:teuthology.orchestra.run.trial153.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:37.794 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:37.794 DEBUG:teuthology.orchestra.run.trial153:> sudo rpm -qi grub2-tools 2026-02-24T16:31:37.863 INFO:teuthology.orchestra.run.trial153.stdout:Name : grub2-tools 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Epoch : 1 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Version : 2.06 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Release : 120.el9 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Architecture: x86_64 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Group : Unspecified 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Size : 9108611 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:License : GPLv3+ 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:31:37.864 INFO:teuthology.orchestra.run.trial153.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Packager : builder@centos.org 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Vendor : CentOS 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Summary : Support tools for GRUB. 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:Description : 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:31:37.865 INFO:teuthology.orchestra.run.trial153.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:31:37.866 INFO:teuthology.orchestra.run.trial153.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:31:37.866 INFO:teuthology.orchestra.run.trial153.stdout:hardware devices. 2026-02-24T16:31:37.866 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:31:37.866 INFO:teuthology.orchestra.run.trial153.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:31:37.867 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:31:37.867 INFO:teuthology.task.kernel:Updating grub on trial153 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:31:37.867 DEBUG:teuthology.orchestra.run.trial153:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:31:37.930 INFO:teuthology.orchestra.run.trial153.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:31:37.932 DEBUG:teuthology.orchestra.run.trial153:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:31:37.995 INFO:teuthology.orchestra.run.trial153.stdout:/sbin/grubby 2026-02-24T16:31:37.997 INFO:teuthology.orchestra.run.trial153.stdout:yes 2026-02-24T16:31:37.997 DEBUG:teuthology.orchestra.run.trial153:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:31:38.060 INFO:teuthology.orchestra.run.trial153.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:31:38.062 DEBUG:teuthology.orchestra.run.trial153:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.127 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:38.133 DEBUG:teuthology.orchestra.run.trial153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:31:38.149 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:38.337 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:38.338 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:38.338 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:38.338 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:38.340 INFO:teuthology.orchestra.run.trial153.stderr:Generating grub configuration file ... 2026-02-24T16:31:38.395 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:38.395 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:38.395 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:31:38.395 INFO:teuthology.orchestra.run.trial052.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.396 INFO:teuthology.orchestra.run.trial052.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.396 INFO:teuthology.orchestra.run.trial052.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.396 INFO:teuthology.orchestra.run.trial052.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.396 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:38.396 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:31:38.407 INFO:teuthology.orchestra.run.trial153.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.414 INFO:teuthology.orchestra.run.trial153.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:31:38.458 DEBUG:teuthology.orchestra.run.trial052:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:38.487 INFO:teuthology.orchestra.run.trial052.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.489 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:38.489 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:31:38.489 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:31:38.489 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:31:38.489 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2026-02-24T16:31:38.489 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial052.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:31:38.489 DEBUG:teuthology.orchestra.run.trial052:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:38.554 INFO:teuthology.orchestra.run.trial052.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.555 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:38.555 DEBUG:teuthology.orchestra.run.trial052:> sudo rpm -qi grub2-tools 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Name : grub2-tools 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Epoch : 1 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Version : 2.06 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Release : 120.el9 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Architecture: x86_64 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Group : Unspecified 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Size : 9108611 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:License : GPLv3+ 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Packager : builder@centos.org 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Vendor : CentOS 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Summary : Support tools for GRUB. 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:Description : 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:hardware devices. 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:31:38.624 INFO:teuthology.orchestra.run.trial052.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:31:38.625 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:31:38.625 INFO:teuthology.task.kernel:Updating grub on trial052 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:31:38.626 DEBUG:teuthology.orchestra.run.trial052:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:31:38.661 INFO:teuthology.orchestra.run.trial153.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:31:38.668 INFO:teuthology.orchestra.run.trial153.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:31:38.688 INFO:teuthology.orchestra.run.trial052.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:31:38.689 DEBUG:teuthology.orchestra.run.trial052:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:31:38.701 INFO:teuthology.orchestra.run.trial153.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:31:38.708 INFO:teuthology.orchestra.run.trial153.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:31:38.739 INFO:teuthology.orchestra.run.trial153.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:31:38.745 INFO:teuthology.orchestra.run.trial153.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:31:38.752 INFO:teuthology.orchestra.run.trial052.stdout:/sbin/grubby 2026-02-24T16:31:38.753 INFO:teuthology.orchestra.run.trial052.stdout:yes 2026-02-24T16:31:38.753 DEBUG:teuthology.orchestra.run.trial052:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:31:38.768 INFO:teuthology.orchestra.run.trial153.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:31:38.775 INFO:teuthology.orchestra.run.trial153.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:31:38.817 INFO:teuthology.orchestra.run.trial052.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:31:38.818 DEBUG:teuthology.orchestra.run.trial052:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:31:38.862 INFO:teuthology.orchestra.run.trial153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:31:38.870 INFO:teuthology.orchestra.run.trial153.stderr:done 2026-02-24T16:31:38.872 DEBUG:teuthology.orchestra.run.trial153:> test -d /sys/firmware/efi 2026-02-24T16:31:38.888 DEBUG:teuthology.orchestra.run.trial153:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:31:38.892 DEBUG:teuthology.orchestra.run.trial052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:31:38.954 INFO:teuthology.orchestra.run.trial153.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:31:38.955 INFO:teuthology.orchestra.run.trial153.stdout:/boot/efi/EFI/centos 2026-02-24T16:31:38.956 DEBUG:teuthology.orchestra.run.trial153:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:31:39.021 INFO:teuthology.orchestra.run.trial153.stdout:no 2026-02-24T16:31:39.022 DEBUG:teuthology.orchestra.run.trial153:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:31:39.087 INFO:teuthology.orchestra.run.trial153.stdout:yes 2026-02-24T16:31:39.087 DEBUG:teuthology.orchestra.run.trial153:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:31:39.104 INFO:teuthology.orchestra.run.trial052.stderr:Generating grub configuration file ... 2026-02-24T16:31:39.152 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial153 2026-02-24T16:31:39.152 DEBUG:teuthology.orchestra.run.trial153:> sudo bash -c ' 2026-02-24T16:31:39.152 DEBUG:teuthology.orchestra.run.trial153:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:31:39.152 DEBUG:teuthology.orchestra.run.trial153:> ' 2026-02-24T16:31:39.171 INFO:teuthology.orchestra.run.trial052.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:31:39.178 INFO:teuthology.orchestra.run.trial052.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:31:39.430 INFO:teuthology.orchestra.run.trial052.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:31:39.436 INFO:teuthology.orchestra.run.trial052.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:31:39.470 INFO:teuthology.orchestra.run.trial052.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:31:39.477 INFO:teuthology.orchestra.run.trial052.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:31:39.508 INFO:teuthology.orchestra.run.trial052.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:31:39.515 INFO:teuthology.orchestra.run.trial052.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:31:39.539 INFO:teuthology.orchestra.run.trial052.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:31:39.545 INFO:teuthology.orchestra.run.trial052.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:31:39.631 INFO:teuthology.orchestra.run.trial052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:31:39.639 INFO:teuthology.orchestra.run.trial052.stderr:done 2026-02-24T16:31:39.641 DEBUG:teuthology.orchestra.run.trial052:> test -d /sys/firmware/efi 2026-02-24T16:31:39.656 DEBUG:teuthology.orchestra.run.trial052:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:31:39.721 INFO:teuthology.orchestra.run.trial052.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:31:39.721 INFO:teuthology.orchestra.run.trial052.stdout:/boot/efi/EFI/centos 2026-02-24T16:31:39.723 DEBUG:teuthology.orchestra.run.trial052:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:31:39.787 INFO:teuthology.orchestra.run.trial052.stdout:no 2026-02-24T16:31:39.788 DEBUG:teuthology.orchestra.run.trial052:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:31:39.851 INFO:teuthology.orchestra.run.trial052.stdout:yes 2026-02-24T16:31:39.852 DEBUG:teuthology.orchestra.run.trial052:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:31:39.864 INFO:teuthology.orchestra.run.trial051.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 957 kB/s | 31 MB 00:33 2026-02-24T16:31:39.867 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:31:39.867 INFO:teuthology.orchestra.run.trial051.stdout:Total 2.5 MB/s | 89 MB 00:34 2026-02-24T16:31:39.916 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial052 2026-02-24T16:31:39.916 DEBUG:teuthology.orchestra.run.trial052:> sudo bash -c ' 2026-02-24T16:31:39.916 DEBUG:teuthology.orchestra.run.trial052:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:31:39.916 DEBUG:teuthology.orchestra.run.trial052:> ' 2026-02-24T16:31:40.058 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:31:40.262 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:31:40.262 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:31:40.648 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:31:40.648 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:31:41.759 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:31:42.048 INFO:teuthology.orchestra.run.trial051.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:42.071 INFO:teuthology.orchestra.run.trial051.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:42.155 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:42.176 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:42.368 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:42.368 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:31:42.368 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:42.368 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:42.426 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:31:42.487 DEBUG:teuthology.orchestra.run.trial183:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:42.517 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.517 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:42.517 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:31:42.517 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:31:42.517 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:31:42.517 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2026-02-24T16:31:42.517 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial183.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:31:42.518 DEBUG:teuthology.orchestra.run.trial183:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:31:42.582 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.583 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:31:42.583 DEBUG:teuthology.orchestra.run.trial183:> sudo rpm -qi grub2-tools 2026-02-24T16:31:42.651 INFO:teuthology.orchestra.run.trial183.stdout:Name : grub2-tools 2026-02-24T16:31:42.651 INFO:teuthology.orchestra.run.trial183.stdout:Epoch : 1 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Version : 2.06 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Release : 120.el9 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Architecture: x86_64 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Group : Unspecified 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Size : 9108611 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:License : GPLv3+ 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Packager : builder@centos.org 2026-02-24T16:31:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Vendor : CentOS 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:Summary : Support tools for GRUB. 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:Description : 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:hardware devices. 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:31:42.653 INFO:teuthology.orchestra.run.trial183.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:31:42.654 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:31:42.654 INFO:teuthology.task.kernel:Updating grub on trial183 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:31:42.654 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:31:42.717 INFO:teuthology.orchestra.run.trial183.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:31:42.718 DEBUG:teuthology.orchestra.run.trial183:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:31:42.781 INFO:teuthology.orchestra.run.trial183.stdout:/sbin/grubby 2026-02-24T16:31:42.782 INFO:teuthology.orchestra.run.trial183.stdout:yes 2026-02-24T16:31:42.782 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:31:42.848 INFO:teuthology.orchestra.run.trial183.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:31:42.850 DEBUG:teuthology.orchestra.run.trial183:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:31:42.922 DEBUG:teuthology.orchestra.run.trial183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:31:43.134 INFO:teuthology.orchestra.run.trial183.stderr:Generating grub configuration file ... 2026-02-24T16:31:43.202 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:31:43.208 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:31:43.458 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:31:43.465 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:31:43.499 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:31:43.506 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:31:43.536 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:31:43.543 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:31:43.566 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:31:43.572 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:31:43.660 INFO:teuthology.orchestra.run.trial183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:31:43.669 INFO:teuthology.orchestra.run.trial183.stderr:done 2026-02-24T16:31:43.671 DEBUG:teuthology.orchestra.run.trial183:> test -d /sys/firmware/efi 2026-02-24T16:31:43.687 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:31:43.753 INFO:teuthology.orchestra.run.trial183.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:31:43.753 INFO:teuthology.orchestra.run.trial183.stdout:/boot/efi/EFI/centos 2026-02-24T16:31:43.754 DEBUG:teuthology.orchestra.run.trial183:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:31:43.820 INFO:teuthology.orchestra.run.trial183.stdout:no 2026-02-24T16:31:43.821 DEBUG:teuthology.orchestra.run.trial183:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:31:43.884 INFO:teuthology.orchestra.run.trial183.stdout:yes 2026-02-24T16:31:43.884 DEBUG:teuthology.orchestra.run.trial183:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:31:43.948 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial183 2026-02-24T16:31:43.948 DEBUG:teuthology.orchestra.run.trial183:> sudo bash -c ' 2026-02-24T16:31:43.948 DEBUG:teuthology.orchestra.run.trial183:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:31:43.948 DEBUG:teuthology.orchestra.run.trial183:> ' 2026-02-24T16:31:44.642 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:31:44.676 INFO:teuthology.orchestra.run.trial051.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:47.777 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:31:47.784 INFO:teuthology.orchestra.run.trial051.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:31:50.833 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:32:07.364 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2026-02-24T16:32:07.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial151.front.sepia.ceph.com' 2026-02-24T16:32:07.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:07.810 INFO:teuthology.task.kernel:Checking client host.d for new kernel version... 2026-02-24T16:32:07.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T16:32:07.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:08.304 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:32:08.325 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:32:08.513 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:32:08.513 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T16:32:08.514 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T16:32:08.514 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.571 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:32:08.572 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:32:08.632 DEBUG:teuthology.orchestra.run.trial051:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:32:08.662 INFO:teuthology.orchestra.run.trial051.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:32:08.662 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T16:32:08.662 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T16:32:08.663 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T16:32:08.663 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2026-02-24T16:32:08.663 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial051.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T16:32:08.663 DEBUG:teuthology.orchestra.run.trial051:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T16:32:08.728 INFO:teuthology.orchestra.run.trial051.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T16:32:08.728 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T16:32:08.729 DEBUG:teuthology.orchestra.run.trial051:> sudo rpm -qi grub2-tools 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Name : grub2-tools 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Epoch : 1 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Version : 2.06 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Release : 120.el9 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Architecture: x86_64 2026-02-24T16:32:08.796 INFO:teuthology.orchestra.run.trial051.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Group : Unspecified 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Size : 9108611 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:License : GPLv3+ 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Packager : builder@centos.org 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Vendor : CentOS 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Summary : Support tools for GRUB. 2026-02-24T16:32:08.797 INFO:teuthology.orchestra.run.trial051.stdout:Description : 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout:hardware devices. 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:32:08.798 INFO:teuthology.orchestra.run.trial051.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T16:32:08.799 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T16:32:08.799 INFO:teuthology.task.kernel:Updating grub on trial051 to boot 5.14.0-686.el9.x86_64 2026-02-24T16:32:08.800 DEBUG:teuthology.orchestra.run.trial051:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T16:32:08.862 INFO:teuthology.orchestra.run.trial051.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T16:32:08.864 DEBUG:teuthology.orchestra.run.trial051:> sudo command -v grubby && echo yes || echo no 2026-02-24T16:32:08.928 INFO:teuthology.orchestra.run.trial051.stdout:/sbin/grubby 2026-02-24T16:32:08.930 INFO:teuthology.orchestra.run.trial051.stdout:yes 2026-02-24T16:32:08.930 DEBUG:teuthology.orchestra.run.trial051:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T16:32:08.993 INFO:teuthology.orchestra.run.trial051.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T16:32:08.994 DEBUG:teuthology.orchestra.run.trial051:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T16:32:09.069 DEBUG:teuthology.orchestra.run.trial051:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T16:32:09.194 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2026-02-24T16:32:09.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-24T16:32:09.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:09.276 INFO:teuthology.orchestra.run.trial051.stderr:Generating grub configuration file ... 2026-02-24T16:32:09.344 INFO:teuthology.orchestra.run.trial051.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T16:32:09.351 INFO:teuthology.orchestra.run.trial051.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T16:32:09.604 INFO:teuthology.orchestra.run.trial051.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T16:32:09.611 INFO:teuthology.orchestra.run.trial051.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T16:32:09.646 INFO:teuthology.orchestra.run.trial051.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T16:32:09.653 INFO:teuthology.orchestra.run.trial051.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T16:32:09.685 INFO:teuthology.orchestra.run.trial051.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T16:32:09.691 INFO:teuthology.orchestra.run.trial051.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T16:32:09.715 INFO:teuthology.orchestra.run.trial051.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T16:32:09.721 INFO:teuthology.orchestra.run.trial051.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T16:32:09.809 INFO:teuthology.orchestra.run.trial051.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T16:32:09.817 INFO:teuthology.orchestra.run.trial051.stderr:done 2026-02-24T16:32:09.819 DEBUG:teuthology.orchestra.run.trial051:> test -d /sys/firmware/efi 2026-02-24T16:32:09.834 DEBUG:teuthology.orchestra.run.trial051:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T16:32:09.899 INFO:teuthology.orchestra.run.trial051.stdout:/boot/efi/EFI/BOOT 2026-02-24T16:32:09.899 INFO:teuthology.orchestra.run.trial051.stdout:/boot/efi/EFI/centos 2026-02-24T16:32:09.900 DEBUG:teuthology.orchestra.run.trial051:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T16:32:09.960 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2026-02-24T16:32:09.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial052.front.sepia.ceph.com' 2026-02-24T16:32:09.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:09.967 INFO:teuthology.orchestra.run.trial051.stdout:no 2026-02-24T16:32:09.968 DEBUG:teuthology.orchestra.run.trial051:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T16:32:10.032 INFO:teuthology.orchestra.run.trial051.stdout:yes 2026-02-24T16:32:10.032 DEBUG:teuthology.orchestra.run.trial051:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T16:32:10.096 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial051 2026-02-24T16:32:10.096 DEBUG:teuthology.orchestra.run.trial051:> sudo bash -c ' 2026-02-24T16:32:10.096 DEBUG:teuthology.orchestra.run.trial051:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T16:32:10.096 DEBUG:teuthology.orchestra.run.trial051:> ' 2026-02-24T16:32:13.991 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2026-02-24T16:32:13.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-24T16:32:13.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:21.632 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.151 2026-02-24T16:32:22.080 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.191 2026-02-24T16:32:23.489 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.153 2026-02-24T16:32:24.256 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.52 2026-02-24T16:32:28.288 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.183 2026-02-24T16:32:30.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial151.front.sepia.ceph.com' 2026-02-24T16:32:30.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:31.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T16:32:31.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:32.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-24T16:32:32.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:33.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial052.front.sepia.ceph.com' 2026-02-24T16:32:33.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:37.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-24T16:32:37.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:40.139 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2026-02-24T16:32:40.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-24T16:32:40.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:32:54.400 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.51 2026-02-24T16:33:03.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-24T16:33:03.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:33:06.559 DEBUG:teuthology.orchestra.run.trial151:> true 2026-02-24T16:33:06.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial151.front.sepia.ceph.com' 2026-02-24T16:33:06.710 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:06.710 DEBUG:teuthology.orchestra.run.trial151:> uname -r 2026-02-24T16:33:06.767 INFO:teuthology.orchestra.run.trial151.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:06.767 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:06.767 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:06.768 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:07.033 DEBUG:teuthology.orchestra.run.trial191:> true 2026-02-24T16:33:07.187 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T16:33:07.188 INFO:teuthology.task.kernel:Checking kernel version of host.d, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:07.188 DEBUG:teuthology.orchestra.run.trial191:> uname -r 2026-02-24T16:33:07.246 INFO:teuthology.orchestra.run.trial191.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:07.247 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:07.247 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:07.247 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:07.768 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2026-02-24T16:33:07.769 DEBUG:teuthology.orchestra.run.trial151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:07.799 INFO:teuthology.orchestra.run.trial151.stdout:ttyS1 2026-02-24T16:33:07.821 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:08.248 INFO:teuthology.task.kernel:Enabling kdb on host.d... 2026-02-24T16:33:08.249 DEBUG:teuthology.orchestra.run.trial191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:08.278 INFO:teuthology.orchestra.run.trial191.stdout:ttyS1 2026-02-24T16:33:08.301 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:08.615 DEBUG:teuthology.orchestra.run.trial153:> true 2026-02-24T16:33:08.772 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-24T16:33:08.773 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:08.773 DEBUG:teuthology.orchestra.run.trial153:> uname -r 2026-02-24T16:33:08.829 INFO:teuthology.orchestra.run.trial153.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:08.829 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:08.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:08.829 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:09.124 DEBUG:teuthology.orchestra.run.trial052:> true 2026-02-24T16:33:09.277 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial052.front.sepia.ceph.com' 2026-02-24T16:33:09.277 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:09.277 DEBUG:teuthology.orchestra.run.trial052:> uname -r 2026-02-24T16:33:09.333 INFO:teuthology.orchestra.run.trial052.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:09.333 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:09.334 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:09.334 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:09.830 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2026-02-24T16:33:09.830 DEBUG:teuthology.orchestra.run.trial153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:09.860 INFO:teuthology.orchestra.run.trial153.stdout:ttyS1 2026-02-24T16:33:09.887 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:10.335 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2026-02-24T16:33:10.335 DEBUG:teuthology.orchestra.run.trial052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:10.366 INFO:teuthology.orchestra.run.trial052.stdout:ttyS1 2026-02-24T16:33:10.386 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:13.192 DEBUG:teuthology.orchestra.run.trial183:> true 2026-02-24T16:33:13.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-24T16:33:13.357 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:13.358 DEBUG:teuthology.orchestra.run.trial183:> uname -r 2026-02-24T16:33:13.413 INFO:teuthology.orchestra.run.trial183.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:13.413 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:13.413 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:13.413 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:14.414 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2026-02-24T16:33:14.414 DEBUG:teuthology.orchestra.run.trial183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:14.445 INFO:teuthology.orchestra.run.trial183.stdout:ttyS1 2026-02-24T16:33:14.469 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:39.313 DEBUG:teuthology.orchestra.run.trial051:> true 2026-02-24T16:33:39.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-24T16:33:39.472 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-686.el9.x86_64"... 2026-02-24T16:33:39.472 DEBUG:teuthology.orchestra.run.trial051:> uname -r 2026-02-24T16:33:39.528 INFO:teuthology.orchestra.run.trial051.stdout:5.14.0-686.el9.x86_64 2026-02-24T16:33:39.529 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T16:33:39.529 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T16:33:39.529 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T16:33:40.530 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2026-02-24T16:33:40.531 DEBUG:teuthology.orchestra.run.trial051:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T16:33:40.561 INFO:teuthology.orchestra.run.trial051.stdout:ttyS1 2026-02-24T16:33:40.582 DEBUG:teuthology.parallel:result is None 2026-02-24T16:33:40.582 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T16:33:40.595 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T16:33:40.595 DEBUG:teuthology.orchestra.run.trial051:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.623 DEBUG:teuthology.orchestra.run.trial052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.625 DEBUG:teuthology.orchestra.run.trial151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.627 DEBUG:teuthology.orchestra.run.trial153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.629 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.631 DEBUG:teuthology.orchestra.run.trial191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T16:33:40.649 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T16:33:40.664 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T16:33:40.676 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T16:33:40.676 DEBUG:teuthology.orchestra.run.trial051:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.678 DEBUG:teuthology.orchestra.run.trial052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.682 DEBUG:teuthology.orchestra.run.trial151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.685 DEBUG:teuthology.orchestra.run.trial153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.687 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.689 DEBUG:teuthology.orchestra.run.trial191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T16:33:40.714 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T16:33:40.725 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T16:33:40.725 DEBUG:teuthology.orchestra.run.trial051:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.735 DEBUG:teuthology.orchestra.run.trial052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.738 DEBUG:teuthology.orchestra.run.trial151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.741 DEBUG:teuthology.orchestra.run.trial153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.743 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.745 DEBUG:teuthology.orchestra.run.trial191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T16:33:40.757 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.761 INFO:teuthology.orchestra.run.trial052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.763 INFO:teuthology.orchestra.run.trial151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.765 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.765 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.768 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.770 INFO:teuthology.orchestra.run.trial052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.772 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.772 INFO:teuthology.orchestra.run.trial151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.775 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.777 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.780 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T16:33:40.782 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T16:33:40.794 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T16:33:40.794 DEBUG:teuthology.orchestra.run.trial051:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.809 DEBUG:teuthology.orchestra.run.trial052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.813 DEBUG:teuthology.orchestra.run.trial151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.816 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.818 DEBUG:teuthology.orchestra.run.trial183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.819 DEBUG:teuthology.orchestra.run.trial191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T16:33:40.845 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T16:33:40.860 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T16:33:40.860 DEBUG:teuthology.orchestra.run.trial051:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.872 DEBUG:teuthology.orchestra.run.trial052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.877 DEBUG:teuthology.orchestra.run.trial151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.880 DEBUG:teuthology.orchestra.run.trial153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.882 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.883 DEBUG:teuthology.orchestra.run.trial191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T16:33:40.901 DEBUG:teuthology.orchestra.run.trial051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.005 DEBUG:teuthology.orchestra.run.trial051:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.029 DEBUG:teuthology.orchestra.run.trial051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.143 DEBUG:teuthology.orchestra.run.trial051:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.166 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:33:41.166 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:41.230 DEBUG:teuthology.orchestra.run.trial052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.311 DEBUG:teuthology.orchestra.run.trial052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.337 DEBUG:teuthology.orchestra.run.trial052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.483 DEBUG:teuthology.orchestra.run.trial052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.508 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:33:41.508 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:41.573 DEBUG:teuthology.orchestra.run.trial151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.684 DEBUG:teuthology.orchestra.run.trial151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:41.709 DEBUG:teuthology.orchestra.run.trial151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.837 DEBUG:teuthology.orchestra.run.trial151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:41.862 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:33:41.862 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:41.927 DEBUG:teuthology.orchestra.run.trial153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.040 DEBUG:teuthology.orchestra.run.trial153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.066 DEBUG:teuthology.orchestra.run.trial153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.199 DEBUG:teuthology.orchestra.run.trial153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.224 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:33:42.224 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:42.288 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.403 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.429 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.549 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.575 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:33:42.575 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:42.639 DEBUG:teuthology.orchestra.run.trial191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.724 DEBUG:teuthology.orchestra.run.trial191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:33:42.748 DEBUG:teuthology.orchestra.run.trial191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.890 DEBUG:teuthology.orchestra.run.trial191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:33:42.914 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:33:42.914 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T16:33:42.977 DEBUG:teuthology.orchestra.run.trial051:> sudo service rsyslog restart 2026-02-24T16:33:42.979 DEBUG:teuthology.orchestra.run.trial052:> sudo service rsyslog restart 2026-02-24T16:33:42.981 DEBUG:teuthology.orchestra.run.trial151:> sudo service rsyslog restart 2026-02-24T16:33:42.984 DEBUG:teuthology.orchestra.run.trial153:> sudo service rsyslog restart 2026-02-24T16:33:42.986 DEBUG:teuthology.orchestra.run.trial183:> sudo service rsyslog restart 2026-02-24T16:33:42.989 DEBUG:teuthology.orchestra.run.trial191:> sudo service rsyslog restart 2026-02-24T16:33:43.006 INFO:teuthology.orchestra.run.trial051.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.008 INFO:teuthology.orchestra.run.trial052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.010 INFO:teuthology.orchestra.run.trial151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.012 INFO:teuthology.orchestra.run.trial153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.015 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.044 INFO:teuthology.orchestra.run.trial191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:33:43.530 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T16:33:43.547 INFO:teuthology.task.internal:Starting timer... 2026-02-24T16:33:43.547 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T16:33:43.572 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T16:33:43.591 DEBUG:teuthology.orchestra.run.trial051:> sudo service auditd rotate 2026-02-24T16:33:43.620 INFO:teuthology.orchestra.run.trial051.stdout:Rotating logs: 2026-02-24T16:33:43.622 DEBUG:teuthology.orchestra.run.trial052:> sudo service auditd rotate 2026-02-24T16:33:43.652 INFO:teuthology.orchestra.run.trial052.stdout:Rotating logs: 2026-02-24T16:33:43.653 DEBUG:teuthology.orchestra.run.trial151:> sudo service auditd rotate 2026-02-24T16:33:43.684 INFO:teuthology.orchestra.run.trial151.stdout:Rotating logs: 2026-02-24T16:33:43.685 DEBUG:teuthology.orchestra.run.trial153:> sudo service auditd rotate 2026-02-24T16:33:43.714 INFO:teuthology.orchestra.run.trial153.stdout:Rotating logs: 2026-02-24T16:33:43.715 DEBUG:teuthology.orchestra.run.trial183:> sudo service auditd rotate 2026-02-24T16:33:43.745 INFO:teuthology.orchestra.run.trial183.stdout:Rotating logs: 2026-02-24T16:33:43.747 DEBUG:teuthology.orchestra.run.trial191:> sudo service auditd rotate 2026-02-24T16:33:43.777 INFO:teuthology.orchestra.run.trial191.stdout:Rotating logs: 2026-02-24T16:33:43.779 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T16:33:43.779 DEBUG:teuthology.orchestra.run.trial051:> /usr/sbin/getenforce 2026-02-24T16:33:43.797 INFO:teuthology.orchestra.run.trial051.stdout:Permissive 2026-02-24T16:33:43.798 DEBUG:teuthology.orchestra.run.trial052:> /usr/sbin/getenforce 2026-02-24T16:33:43.815 INFO:teuthology.orchestra.run.trial052.stdout:Permissive 2026-02-24T16:33:43.816 DEBUG:teuthology.orchestra.run.trial151:> /usr/sbin/getenforce 2026-02-24T16:33:43.834 INFO:teuthology.orchestra.run.trial151.stdout:Permissive 2026-02-24T16:33:43.834 DEBUG:teuthology.orchestra.run.trial153:> /usr/sbin/getenforce 2026-02-24T16:33:43.852 INFO:teuthology.orchestra.run.trial153.stdout:Permissive 2026-02-24T16:33:43.853 DEBUG:teuthology.orchestra.run.trial183:> /usr/sbin/getenforce 2026-02-24T16:33:43.870 INFO:teuthology.orchestra.run.trial183.stdout:Permissive 2026-02-24T16:33:43.871 DEBUG:teuthology.orchestra.run.trial191:> /usr/sbin/getenforce 2026-02-24T16:33:43.888 INFO:teuthology.orchestra.run.trial191.stdout:Permissive 2026-02-24T16:33:43.889 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial051.front.sepia.ceph.com': 'permissive', 'ubuntu@trial052.front.sepia.ceph.com': 'permissive', 'ubuntu@trial151.front.sepia.ceph.com': 'permissive', 'ubuntu@trial153.front.sepia.ceph.com': 'permissive', 'ubuntu@trial183.front.sepia.ceph.com': 'permissive', 'ubuntu@trial191.front.sepia.ceph.com': 'permissive'} 2026-02-24T16:33:43.889 DEBUG:teuthology.orchestra.run.trial051:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:43.915 DEBUG:teuthology.orchestra.run.trial052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:43.941 DEBUG:teuthology.orchestra.run.trial151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:43.967 DEBUG:teuthology.orchestra.run.trial153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:43.995 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:44.021 DEBUG:teuthology.orchestra.run.trial191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:33:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:33:44.049 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T16:33:44.049 DEBUG:teuthology.orchestra.run.trial051:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.075 DEBUG:teuthology.orchestra.run.trial052:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.100 DEBUG:teuthology.orchestra.run.trial151:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.125 DEBUG:teuthology.orchestra.run.trial153:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.149 DEBUG:teuthology.orchestra.run.trial183:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.174 DEBUG:teuthology.orchestra.run.trial191:> sudo /usr/sbin/setenforce permissive 2026-02-24T16:33:44.202 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T16:33:44.218 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-24T16:33:44.220 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-24T16:33:44.360 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T16:33:44.371 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T16:33:44.373 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial051.front.sepia.ceph.com,trial052.front.sepia.ceph.com,trial151.front.sepia.ceph.com,trial153.front.sepia.ceph.com,trial183.front.sepia.ceph.com,trial191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T16:37:33.230 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial051.front.sepia.ceph.com'), Remote(name='ubuntu@trial052.front.sepia.ceph.com'), Remote(name='ubuntu@trial151.front.sepia.ceph.com'), Remote(name='ubuntu@trial153.front.sepia.ceph.com'), Remote(name='ubuntu@trial183.front.sepia.ceph.com'), Remote(name='ubuntu@trial191.front.sepia.ceph.com')] 2026-02-24T16:37:33.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-24T16:37:33.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:33.324 DEBUG:teuthology.orchestra.run.trial051:> true 2026-02-24T16:37:33.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-24T16:37:33.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial052.front.sepia.ceph.com' 2026-02-24T16:37:33.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:33.482 DEBUG:teuthology.orchestra.run.trial052:> true 2026-02-24T16:37:33.556 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial052.front.sepia.ceph.com' 2026-02-24T16:37:33.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial151.front.sepia.ceph.com' 2026-02-24T16:37:33.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:33.640 DEBUG:teuthology.orchestra.run.trial151:> true 2026-02-24T16:37:33.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial151.front.sepia.ceph.com' 2026-02-24T16:37:33.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-24T16:37:33.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:33.802 DEBUG:teuthology.orchestra.run.trial153:> true 2026-02-24T16:37:33.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-24T16:37:33.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-24T16:37:33.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:33.977 DEBUG:teuthology.orchestra.run.trial183:> true 2026-02-24T16:37:34.053 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-24T16:37:34.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T16:37:34.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T16:37:34.132 DEBUG:teuthology.orchestra.run.trial191:> true 2026-02-24T16:37:34.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T16:37:34.208 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T16:37:34.221 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T16:37:34.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.221 DEBUG:teuthology.orchestra.run.trial051:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.223 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.223 DEBUG:teuthology.orchestra.run.trial052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.226 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.226 DEBUG:teuthology.orchestra.run.trial151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.229 DEBUG:teuthology.orchestra.run.trial153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.231 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.231 DEBUG:teuthology.orchestra.run.trial183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.235 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T16:37:34.235 DEBUG:teuthology.orchestra.run.trial191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:37:34.250 INFO:teuthology.orchestra.run.trial051.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.251 INFO:teuthology.orchestra.run.trial052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.255 INFO:teuthology.orchestra.run.trial151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.258 INFO:teuthology.orchestra.run.trial153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.261 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.262 INFO:teuthology.orchestra.run.trial051.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.263 INFO:teuthology.orchestra.run.trial052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.266 INFO:teuthology.orchestra.run.trial151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.270 INFO:teuthology.orchestra.run.trial153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.273 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.274 INFO:teuthology.orchestra.run.trial191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T16:37:34.283 INFO:teuthology.orchestra.run.trial051.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.284 INFO:teuthology.orchestra.run.trial052.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.286 INFO:teuthology.orchestra.run.trial191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T16:37:34.286 INFO:teuthology.orchestra.run.trial151.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.291 INFO:teuthology.orchestra.run.trial153.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.292 INFO:teuthology.orchestra.run.trial051.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.293 INFO:teuthology.orchestra.run.trial052.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.296 INFO:teuthology.orchestra.run.trial183.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.297 INFO:teuthology.orchestra.run.trial151.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.300 INFO:teuthology.orchestra.run.trial153.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.304 INFO:teuthology.orchestra.run.trial051.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.305 INFO:teuthology.orchestra.run.trial183.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.305 INFO:teuthology.orchestra.run.trial052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.306 INFO:teuthology.orchestra.run.trial191.stderr:sudo: ntpd: command not found 2026-02-24T16:37:34.308 INFO:teuthology.orchestra.run.trial151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.312 INFO:teuthology.orchestra.run.trial153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.316 INFO:teuthology.orchestra.run.trial191.stdout:506 Cannot talk to daemon 2026-02-24T16:37:34.317 INFO:teuthology.orchestra.run.trial051.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.317 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.317 INFO:teuthology.orchestra.run.trial052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.319 INFO:teuthology.orchestra.run.trial151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.324 INFO:teuthology.orchestra.run.trial153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.328 INFO:teuthology.orchestra.run.trial191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T16:37:34.329 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.340 INFO:teuthology.orchestra.run.trial191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T16:37:34.366 INFO:teuthology.orchestra.run.trial051.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.366 INFO:teuthology.orchestra.run.trial151.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.367 INFO:teuthology.orchestra.run.trial052.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.368 INFO:teuthology.orchestra.run.trial051.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.368 INFO:teuthology.orchestra.run.trial051.stdout:=============================================================================== 2026-02-24T16:37:34.370 INFO:teuthology.orchestra.run.trial151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.370 INFO:teuthology.orchestra.run.trial151.stdout:=============================================================================== 2026-02-24T16:37:34.370 INFO:teuthology.orchestra.run.trial052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.371 INFO:teuthology.orchestra.run.trial052.stdout:=============================================================================== 2026-02-24T16:37:34.371 INFO:teuthology.orchestra.run.trial153.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.373 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.385 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.385 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-24T16:37:34.385 INFO:teuthology.orchestra.run.trial183.stdout:^? h134-215-155-177.mdtnwi.> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.385 INFO:teuthology.orchestra.run.trial183.stdout:^? t2.time.bf1.yahoo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.385 INFO:teuthology.orchestra.run.trial153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.386 INFO:teuthology.orchestra.run.trial153.stdout:=============================================================================== 2026-02-24T16:37:34.386 INFO:teuthology.orchestra.run.trial153.stdout:^? h134-215-155-177.mdtnwi.> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.386 INFO:teuthology.orchestra.run.trial153.stdout:^? t2.time.bf1.yahoo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.392 INFO:teuthology.orchestra.run.trial191.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:37:34.408 INFO:teuthology.orchestra.run.trial191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:37:34.408 INFO:teuthology.orchestra.run.trial191.stdout:=============================================================================== 2026-02-24T16:37:34.409 INFO:teuthology.orchestra.run.trial191.stdout:^? 50.218.103.254 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.409 INFO:teuthology.orchestra.run.trial191.stdout:^? t2.time.bf1.yahoo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.409 INFO:teuthology.orchestra.run.trial191.stdout:^? time1.lshiy.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.409 INFO:teuthology.orchestra.run.trial191.stdout:^? 144.202.62.209.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T16:37:34.409 INFO:teuthology.run_tasks:Running task install... 2026-02-24T16:37:34.421 DEBUG:teuthology.task.install:project ceph 2026-02-24T16:37:34.421 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'}} 2026-02-24T16:37:34.421 DEBUG:teuthology.task.install:config {'extra_packages': ['nvme-cli'], 'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'} 2026-02-24T16:37:34.422 INFO:teuthology.task.install:Using flavor: default 2026-02-24T16:37:34.426 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T16:37:34.426 INFO:teuthology.task.install:extra packages: ['nvme-cli'] 2026-02-24T16:37:34.427 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.430 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.430 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.431 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.432 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'tag': None, 'wait_for_package': False} 2026-02-24T16:37:34.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:37:34.773 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.774 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:34.776 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.776 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:34.776 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.776 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:34.779 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.779 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:34.779 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.779 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:34.780 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/ 2026-02-24T16:37:34.780 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5507.g618c318a 2026-02-24T16:37:35.086 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.086 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:37:35.086 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:37:35.090 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.094 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.094 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:37:35.094 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.097 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:37:35.097 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:37:35.097 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:37:35.115 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.115 DEBUG:teuthology.orchestra.run.trial151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.116 DEBUG:teuthology.orchestra.run.trial191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.116 DEBUG:teuthology.orchestra.run.trial153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.117 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.117 DEBUG:teuthology.orchestra.run.trial183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.123 DEBUG:teuthology.orchestra.run.trial052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.124 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2026-02-24T16:37:35.124 DEBUG:teuthology.orchestra.run.trial051:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/618c318a7a0b287ab5aeff207b7bf1d73ef45044/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:37:35.179 DEBUG:teuthology.orchestra.run.trial151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.180 DEBUG:teuthology.orchestra.run.trial191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.181 DEBUG:teuthology.orchestra.run.trial183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.182 DEBUG:teuthology.orchestra.run.trial153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.187 DEBUG:teuthology.orchestra.run.trial052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.187 DEBUG:teuthology.orchestra.run.trial051:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:37:35.252 DEBUG:teuthology.orchestra.run.trial151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.254 DEBUG:teuthology.orchestra.run.trial191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.256 DEBUG:teuthology.orchestra.run.trial153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.256 DEBUG:teuthology.orchestra.run.trial183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.260 DEBUG:teuthology.orchestra.run.trial052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.262 DEBUG:teuthology.orchestra.run.trial051:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:37:35.316 INFO:teuthology.orchestra.run.trial151.stdout:check_obsoletes = 1 2026-02-24T16:37:35.317 DEBUG:teuthology.orchestra.run.trial151:> sudo yum clean all 2026-02-24T16:37:35.318 INFO:teuthology.orchestra.run.trial191.stdout:check_obsoletes = 1 2026-02-24T16:37:35.319 INFO:teuthology.orchestra.run.trial183.stdout:check_obsoletes = 1 2026-02-24T16:37:35.319 DEBUG:teuthology.orchestra.run.trial191:> sudo yum clean all 2026-02-24T16:37:35.320 DEBUG:teuthology.orchestra.run.trial183:> sudo yum clean all 2026-02-24T16:37:35.320 INFO:teuthology.orchestra.run.trial153.stdout:check_obsoletes = 1 2026-02-24T16:37:35.321 DEBUG:teuthology.orchestra.run.trial153:> sudo yum clean all 2026-02-24T16:37:35.324 INFO:teuthology.orchestra.run.trial052.stdout:check_obsoletes = 1 2026-02-24T16:37:35.325 INFO:teuthology.orchestra.run.trial051.stdout:check_obsoletes = 1 2026-02-24T16:37:35.326 DEBUG:teuthology.orchestra.run.trial052:> sudo yum clean all 2026-02-24T16:37:35.326 DEBUG:teuthology.orchestra.run.trial051:> sudo yum clean all 2026-02-24T16:37:35.464 INFO:teuthology.orchestra.run.trial151.stdout:41 files removed 2026-02-24T16:37:35.467 INFO:teuthology.orchestra.run.trial191.stdout:41 files removed 2026-02-24T16:37:35.469 INFO:teuthology.orchestra.run.trial153.stdout:41 files removed 2026-02-24T16:37:35.472 INFO:teuthology.orchestra.run.trial183.stdout:41 files removed 2026-02-24T16:37:35.474 INFO:teuthology.orchestra.run.trial051.stdout:41 files removed 2026-02-24T16:37:35.477 INFO:teuthology.orchestra.run.trial052.stdout:41 files removed 2026-02-24T16:37:35.478 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.481 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.483 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.487 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.489 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.493 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-radosgw 2026-02-24T16:37:35.758 INFO:teuthology.orchestra.run.trial151.stdout:ceph packages for x86_64 717 kB/s | 94 kB 00:00 2026-02-24T16:37:35.758 INFO:teuthology.orchestra.run.trial153.stdout:ceph packages for x86_64 745 kB/s | 94 kB 00:00 2026-02-24T16:37:35.760 INFO:teuthology.orchestra.run.trial191.stdout:ceph packages for x86_64 722 kB/s | 94 kB 00:00 2026-02-24T16:37:35.768 INFO:teuthology.orchestra.run.trial052.stdout:ceph packages for x86_64 738 kB/s | 94 kB 00:00 2026-02-24T16:37:35.770 INFO:teuthology.orchestra.run.trial183.stdout:ceph packages for x86_64 697 kB/s | 94 kB 00:00 2026-02-24T16:37:35.770 INFO:teuthology.orchestra.run.trial051.stdout:ceph packages for x86_64 707 kB/s | 94 kB 00:00 2026-02-24T16:37:35.901 INFO:teuthology.orchestra.run.trial151.stdout:ceph noarch packages 213 kB/s | 24 kB 00:00 2026-02-24T16:37:35.904 INFO:teuthology.orchestra.run.trial153.stdout:ceph noarch packages 208 kB/s | 24 kB 00:00 2026-02-24T16:37:35.906 INFO:teuthology.orchestra.run.trial191.stdout:ceph noarch packages 208 kB/s | 24 kB 00:00 2026-02-24T16:37:35.911 INFO:teuthology.orchestra.run.trial052.stdout:ceph noarch packages 212 kB/s | 24 kB 00:00 2026-02-24T16:37:35.913 INFO:teuthology.orchestra.run.trial051.stdout:ceph noarch packages 214 kB/s | 24 kB 00:00 2026-02-24T16:37:35.915 INFO:teuthology.orchestra.run.trial183.stdout:ceph noarch packages 216 kB/s | 24 kB 00:00 2026-02-24T16:37:36.033 INFO:teuthology.orchestra.run.trial151.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.033 INFO:teuthology.orchestra.run.trial153.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.035 INFO:teuthology.orchestra.run.trial191.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.038 INFO:teuthology.orchestra.run.trial052.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.042 INFO:teuthology.orchestra.run.trial051.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.044 INFO:teuthology.orchestra.run.trial183.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-24T16:37:36.758 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-24T16:37:36.767 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-24T16:37:36.880 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-24T16:37:36.961 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.9 MB 00:00 2026-02-24T16:37:37.753 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.9 MB 00:01 2026-02-24T16:37:38.373 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - AppStream 25 MB/s | 27 MB 00:01 2026-02-24T16:37:40.338 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 27 MB 00:02 2026-02-24T16:37:41.122 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 27 MB 00:03 2026-02-24T16:37:42.994 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 8.0 MB 00:02 2026-02-24T16:37:44.855 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 8.0 MB 00:01 2026-02-24T16:37:45.195 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 8.0 MB 00:02 2026-02-24T16:37:45.554 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - BaseOS 959 kB/s | 8.9 MB 00:09 2026-02-24T16:37:45.612 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 20 kB 00:01 2026-02-24T16:37:46.829 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 20 kB 00:01 2026-02-24T16:37:47.142 INFO:teuthology.orchestra.run.trial052.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 20 kB 00:01 2026-02-24T16:37:47.308 INFO:teuthology.orchestra.run.trial191.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2026-02-24T16:37:47.572 INFO:teuthology.orchestra.run.trial052.stdout:Extra Packages for Enterprise Linux 55 MB/s | 20 MB 00:00 2026-02-24T16:37:48.231 INFO:teuthology.orchestra.run.trial051.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2026-02-24T16:37:49.993 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 27 MB 00:03 2026-02-24T16:37:50.565 INFO:teuthology.orchestra.run.trial191.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-24T16:37:50.728 INFO:teuthology.orchestra.run.trial052.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T16:37:51.488 INFO:teuthology.orchestra.run.trial051.stdout:lab-extras 2.2 MB/s | 50 kB 00:00 2026-02-24T16:37:51.782 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:37:51.783 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:37:51.783 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout:Upgrading: 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:37:51.784 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:37:51.785 INFO:teuthology.orchestra.run.trial191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:37:51.786 INFO:teuthology.orchestra.run.trial191.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout:Installing weak dependencies: 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:37:51.787 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:37:51.788 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:37:51.788 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:37:51.788 INFO:teuthology.orchestra.run.trial191.stdout:Install 35 Packages 2026-02-24T16:37:51.788 INFO:teuthology.orchestra.run.trial191.stdout:Upgrade 2 Packages 2026-02-24T16:37:51.788 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:37:51.789 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 68 M 2026-02-24T16:37:51.789 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:37:51.946 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:37:51.947 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:37:51.948 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repository Size 2026-02-24T16:37:51.948 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:37:51.948 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:37:51.948 INFO:teuthology.orchestra.run.trial052.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout:Upgrading: 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout:Installing dependencies: 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:37:51.949 INFO:teuthology.orchestra.run.trial052.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:37:51.950 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:37:51.951 INFO:teuthology.orchestra.run.trial052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:37:51.952 INFO:teuthology.orchestra.run.trial052.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:37:51.952 INFO:teuthology.orchestra.run.trial052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:37:51.952 INFO:teuthology.orchestra.run.trial052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:37:51.952 INFO:teuthology.orchestra.run.trial052.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:37:51.952 INFO:teuthology.orchestra.run.trial052.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:37:51.953 INFO:teuthology.orchestra.run.trial052.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout:Installing weak dependencies: 2026-02-24T16:37:51.954 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:37:51.955 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:37:51.956 INFO:teuthology.orchestra.run.trial052.stdout:Install 35 Packages 2026-02-24T16:37:51.956 INFO:teuthology.orchestra.run.trial052.stdout:Upgrade 2 Packages 2026-02-24T16:37:51.956 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:37:51.957 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 68 M 2026-02-24T16:37:51.957 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:37:52.434 INFO:teuthology.orchestra.run.trial191.stdout:(1/37): ceph-base-20.3.0-5507.g618c318a.el9.x86 12 MB/s | 5.2 MB 00:00 2026-02-24T16:37:52.489 INFO:teuthology.orchestra.run.trial191.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 458 kB/s | 25 kB 00:00 2026-02-24T16:37:52.545 INFO:teuthology.orchestra.run.trial191.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 687 kB/s | 39 kB 00:00 2026-02-24T16:37:52.598 INFO:teuthology.orchestra.run.trial191.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 498 kB/s | 26 kB 00:00 2026-02-24T16:37:52.682 INFO:teuthology.orchestra.run.trial191.stdout:(5/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 19 MB/s | 13 MB 00:00 2026-02-24T16:37:52.684 INFO:teuthology.orchestra.run.trial191.stdout:(6/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 10 MB/s | 909 kB 00:00 2026-02-24T16:37:52.700 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout:Upgrading: 2026-02-24T16:37:52.702 INFO:teuthology.orchestra.run.trial051.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:37:52.703 INFO:teuthology.orchestra.run.trial051.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:37:52.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout:Installing weak dependencies: 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:37:52.705 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout:Install 35 Packages 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout:Upgrade 2 Packages 2026-02-24T16:37:52.706 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:37:52.707 INFO:teuthology.orchestra.run.trial191.stdout:(7/37): libradosstriper1-20.3.0-5507.g618c318a. 10 MB/s | 248 kB 00:00 2026-02-24T16:37:52.707 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 68 M 2026-02-24T16:37:52.707 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:37:52.741 INFO:teuthology.orchestra.run.trial191.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 1.3 MB/s | 45 kB 00:00 2026-02-24T16:37:52.779 INFO:teuthology.orchestra.run.trial191.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 4.7 MB/s | 186 kB 00:00 2026-02-24T16:37:52.818 INFO:teuthology.orchestra.run.trial191.stdout:(10/37): python3-cephfs-20.3.0-5507.g618c318a.e 5.1 MB/s | 204 kB 00:00 2026-02-24T16:37:52.861 INFO:teuthology.orchestra.run.trial191.stdout:(11/37): python3-rados-20.3.0-5507.g618c318a.el 7.5 MB/s | 324 kB 00:00 2026-02-24T16:37:52.904 INFO:teuthology.orchestra.run.trial191.stdout:(12/37): python3-rbd-20.3.0-5507.g618c318a.el9. 6.9 MB/s | 305 kB 00:00 2026-02-24T16:37:52.945 INFO:teuthology.orchestra.run.trial191.stdout:(13/37): python3-rgw-20.3.0-5507.g618c318a.el9. 2.4 MB/s | 100 kB 00:00 2026-02-24T16:37:52.977 INFO:teuthology.orchestra.run.trial191.stdout:(14/37): ceph-common-20.3.0-5507.g618c318a.el9. 24 MB/s | 24 MB 00:00 2026-02-24T16:37:53.019 INFO:teuthology.orchestra.run.trial191.stdout:(15/37): librgw2-20.3.0-5507.g618c318a.el9.x86_ 19 MB/s | 6.5 MB 00:00 2026-02-24T16:37:53.380 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 8.0 MB 00:01 2026-02-24T16:37:53.389 INFO:teuthology.orchestra.run.trial191.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 75 kB/s | 33 kB 00:00 2026-02-24T16:37:53.453 INFO:teuthology.orchestra.run.trial191.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 1.6 MB/s | 104 kB 00:00 2026-02-24T16:37:53.476 INFO:teuthology.orchestra.run.trial051.stdout:(1/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 23 MB/s | 13 MB 00:00 2026-02-24T16:37:53.496 INFO:teuthology.orchestra.run.trial191.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.7 MB/s | 164 kB 00:00 2026-02-24T16:37:53.497 INFO:teuthology.orchestra.run.trial051.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 1.2 MB/s | 25 kB 00:00 2026-02-24T16:37:53.519 INFO:teuthology.orchestra.run.trial051.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 1.7 MB/s | 39 kB 00:00 2026-02-24T16:37:53.521 INFO:teuthology.orchestra.run.trial191.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.5 MB/s | 160 kB 00:00 2026-02-24T16:37:53.542 INFO:teuthology.orchestra.run.trial051.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 1.1 MB/s | 26 kB 00:00 2026-02-24T16:37:53.542 INFO:teuthology.orchestra.run.trial191.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.0 MB/s | 45 kB 00:00 2026-02-24T16:37:53.575 INFO:teuthology.orchestra.run.trial191.stdout:(21/37): pciutils-3.7.0-7.el9.x86_64.rpm 168 kB/s | 93 kB 00:00 2026-02-24T16:37:53.584 INFO:teuthology.orchestra.run.trial051.stdout:(5/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 21 MB/s | 909 kB 00:00 2026-02-24T16:37:53.588 INFO:teuthology.orchestra.run.trial191.stdout:(22/37): librdkafka-1.6.1-102.el9.x86_64.rpm 14 MB/s | 662 kB 00:00 2026-02-24T16:37:53.611 INFO:teuthology.orchestra.run.trial191.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 10 MB/s | 240 kB 00:00 2026-02-24T16:37:53.615 INFO:teuthology.orchestra.run.trial051.stdout:(6/37): libradosstriper1-20.3.0-5507.g618c318a. 8.0 MB/s | 248 kB 00:00 2026-02-24T16:37:53.633 INFO:teuthology.orchestra.run.trial191.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-24T16:37:53.655 INFO:teuthology.orchestra.run.trial191.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.9 MB/s | 66 kB 00:00 2026-02-24T16:37:53.660 INFO:teuthology.orchestra.run.trial191.stdout:(26/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2026-02-24T16:37:53.715 INFO:teuthology.orchestra.run.trial191.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.1 MB/s | 308 kB 00:00 2026-02-24T16:37:53.725 INFO:teuthology.orchestra.run.trial191.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 2.4 MB/s | 25 kB 00:00 2026-02-24T16:37:53.738 INFO:teuthology.orchestra.run.trial191.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 3.7 MB/s | 49 kB 00:00 2026-02-24T16:37:53.752 INFO:teuthology.orchestra.run.trial191.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 4.8 MB/s | 67 kB 00:00 2026-02-24T16:37:53.791 INFO:teuthology.orchestra.run.trial191.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 21 MB/s | 838 kB 00:00 2026-02-24T16:37:53.804 INFO:teuthology.orchestra.run.trial191.stdout:(32/37): libarrow-9.0.0-15.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2026-02-24T16:37:53.818 INFO:teuthology.orchestra.run.trial191.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 6.9 MB/s | 191 kB 00:00 2026-02-24T16:37:53.846 INFO:teuthology.orchestra.run.trial191.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 38 MB/s | 1.6 MB 00:00 2026-02-24T16:37:53.880 INFO:teuthology.orchestra.run.trial051.stdout:(7/37): librgw2-20.3.0-5507.g618c318a.el9.x86_6 25 MB/s | 6.5 MB 00:00 2026-02-24T16:37:53.921 INFO:teuthology.orchestra.run.trial051.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 1.1 MB/s | 45 kB 00:00 2026-02-24T16:37:53.972 INFO:teuthology.orchestra.run.trial051.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 3.6 MB/s | 186 kB 00:00 2026-02-24T16:37:54.036 INFO:teuthology.orchestra.run.trial051.stdout:(10/37): python3-cephfs-20.3.0-5507.g618c318a.e 3.1 MB/s | 204 kB 00:00 2026-02-24T16:37:54.145 INFO:teuthology.orchestra.run.trial052.stdout:(1/37): ceph-base-20.3.0-5507.g618c318a.el9.x86 3.6 MB/s | 5.2 MB 00:01 2026-02-24T16:37:54.172 INFO:teuthology.orchestra.run.trial191.stdout:(35/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 8.7 MB/s | 2.8 MB 00:00 2026-02-24T16:37:54.187 INFO:teuthology.orchestra.run.trial052.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 593 kB/s | 25 kB 00:00 2026-02-24T16:37:54.196 INFO:teuthology.orchestra.run.trial191.stdout:(36/37): librados2-20.3.0-5507.g618c318a.el9.x8 9.2 MB/s | 3.5 MB 00:00 2026-02-24T16:37:54.217 INFO:teuthology.orchestra.run.trial052.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 1.3 MB/s | 39 kB 00:00 2026-02-24T16:37:54.243 INFO:teuthology.orchestra.run.trial052.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 1.0 MB/s | 26 kB 00:00 2026-02-24T16:37:54.259 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - Extras packages 159 kB/s | 20 kB 00:00 2026-02-24T16:37:54.348 INFO:teuthology.orchestra.run.trial051.stdout:(11/37): python3-rados-20.3.0-5507.g618c318a.el 1.0 MB/s | 324 kB 00:00 2026-02-24T16:37:54.387 INFO:teuthology.orchestra.run.trial051.stdout:(12/37): python3-rbd-20.3.0-5507.g618c318a.el9. 7.7 MB/s | 305 kB 00:00 2026-02-24T16:37:54.400 INFO:teuthology.orchestra.run.trial052.stdout:(5/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 5.6 MB/s | 909 kB 00:00 2026-02-24T16:37:54.419 INFO:teuthology.orchestra.run.trial051.stdout:(13/37): python3-rgw-20.3.0-5507.g618c318a.el9. 3.0 MB/s | 100 kB 00:00 2026-02-24T16:37:54.447 INFO:teuthology.orchestra.run.trial051.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 1.2 MB/s | 33 kB 00:00 2026-02-24T16:37:54.466 INFO:teuthology.orchestra.run.trial052.stdout:(6/37): libradosstriper1-20.3.0-5507.g618c318a. 3.7 MB/s | 248 kB 00:00 2026-02-24T16:37:54.467 INFO:teuthology.orchestra.run.trial051.stdout:(15/37): ceph-base-20.3.0-5507.g618c318a.el9.x8 3.3 MB/s | 5.2 MB 00:01 2026-02-24T16:37:54.497 INFO:teuthology.orchestra.run.trial051.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 23 MB/s | 1.2 MB 00:00 2026-02-24T16:37:54.505 INFO:teuthology.orchestra.run.trial051.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 2.4 MB/s | 93 kB 00:00 2026-02-24T16:37:54.752 INFO:teuthology.orchestra.run.trial191.stdout:(37/37): nvme-cli-2.16-1.el9.x86_64.rpm 664 kB/s | 1.2 MB 00:01 2026-02-24T16:37:54.758 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:37:54.758 INFO:teuthology.orchestra.run.trial191.stdout:Total 23 MB/s | 68 MB 00:02 2026-02-24T16:37:54.857 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:37:54.875 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:37:54.875 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:37:55.062 INFO:teuthology.orchestra.run.trial051.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 184 kB/s | 104 kB 00:00 2026-02-24T16:37:55.143 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:37:55.143 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:37:55.144 INFO:teuthology.orchestra.run.trial051.stdout:(19/37): libnbd-1.20.3-4.el9.x86_64.rpm 256 kB/s | 164 kB 00:00 2026-02-24T16:37:55.228 INFO:teuthology.orchestra.run.trial051.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2026-02-24T16:37:55.302 INFO:teuthology.orchestra.run.trial051.stdout:(21/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 665 kB/s | 160 kB 00:00 2026-02-24T16:37:55.376 INFO:teuthology.orchestra.run.trial052.stdout:(7/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 5.0 MB/s | 13 MB 00:02 2026-02-24T16:37:55.417 INFO:teuthology.orchestra.run.trial052.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 1.1 MB/s | 45 kB 00:00 2026-02-24T16:37:55.456 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:37:55.457 INFO:teuthology.orchestra.run.trial052.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 4.6 MB/s | 186 kB 00:00 2026-02-24T16:37:55.499 INFO:teuthology.orchestra.run.trial052.stdout:(10/37): librgw2-20.3.0-5507.g618c318a.el9.x86_ 6.3 MB/s | 6.5 MB 00:01 2026-02-24T16:37:55.500 INFO:teuthology.orchestra.run.trial052.stdout:(11/37): python3-cephfs-20.3.0-5507.g618c318a.e 4.7 MB/s | 204 kB 00:00 2026-02-24T16:37:55.581 INFO:teuthology.orchestra.run.trial052.stdout:(12/37): python3-rados-20.3.0-5507.g618c318a.el 3.9 MB/s | 324 kB 00:00 2026-02-24T16:37:55.582 INFO:teuthology.orchestra.run.trial052.stdout:(13/37): python3-rbd-20.3.0-5507.g618c318a.el9. 3.6 MB/s | 305 kB 00:00 2026-02-24T16:37:55.608 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:37:55.612 INFO:teuthology.orchestra.run.trial191.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:55.623 INFO:teuthology.orchestra.run.trial052.stdout:(14/37): python3-rgw-20.3.0-5507.g618c318a.el9. 2.3 MB/s | 100 kB 00:00 2026-02-24T16:37:55.661 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:55.662 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:55.675 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:55.678 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:37:55.678 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:55.700 INFO:teuthology.orchestra.run.trial051.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 736 kB/s | 292 kB 00:00 2026-02-24T16:37:55.707 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:55.709 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:37:55.717 INFO:teuthology.orchestra.run.trial191.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:37:55.720 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:37:55.723 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:37:55.727 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:37:55.738 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:37:55.749 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:37:55.750 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:55.773 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:55.777 INFO:teuthology.orchestra.run.trial191.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:37:55.808 INFO:teuthology.orchestra.run.trial191.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:37:55.811 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:37:55.819 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:37:55.821 INFO:teuthology.orchestra.run.trial191.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:37:55.825 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:37:55.948 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:37:55.949 INFO:teuthology.orchestra.run.trial191.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:55.976 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:55.978 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:37:55.985 INFO:teuthology.orchestra.run.trial191.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:37:56.000 INFO:teuthology.orchestra.run.trial153.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2026-02-24T16:37:56.019 INFO:teuthology.orchestra.run.trial051.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 751 kB/s | 240 kB 00:00 2026-02-24T16:37:56.020 INFO:teuthology.orchestra.run.trial051.stdout:(24/37): librdkafka-1.6.1-102.el9.x86_64.rpm 836 kB/s | 662 kB 00:00 2026-02-24T16:37:56.102 INFO:teuthology.orchestra.run.trial051.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 442 kB/s | 37 kB 00:00 2026-02-24T16:37:56.107 INFO:teuthology.orchestra.run.trial051.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 763 kB/s | 66 kB 00:00 2026-02-24T16:37:56.145 INFO:teuthology.orchestra.run.trial052.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 59 kB/s | 33 kB 00:00 2026-02-24T16:37:56.260 INFO:teuthology.orchestra.run.trial051.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2026-02-24T16:37:56.271 INFO:teuthology.orchestra.run.trial191.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:37:56.272 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:56.280 INFO:teuthology.orchestra.run.trial051.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.2 MB/s | 25 kB 00:00 2026-02-24T16:37:56.288 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:56.292 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:37:56.296 INFO:teuthology.orchestra.run.trial191.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:37:56.299 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:37:56.299 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:56.301 INFO:teuthology.orchestra.run.trial051.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 2.3 MB/s | 49 kB 00:00 2026-02-24T16:37:56.316 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:56.323 INFO:teuthology.orchestra.run.trial051.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2026-02-24T16:37:56.350 INFO:teuthology.orchestra.run.trial051.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2026-02-24T16:37:56.372 INFO:teuthology.orchestra.run.trial051.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 8.6 MB/s | 191 kB 00:00 2026-02-24T16:37:56.382 INFO:teuthology.orchestra.run.trial051.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2026-02-24T16:37:56.411 INFO:teuthology.orchestra.run.trial051.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 41 MB/s | 1.6 MB 00:00 2026-02-24T16:37:56.429 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:37:56.431 INFO:teuthology.orchestra.run.trial191.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:56.566 INFO:teuthology.orchestra.run.trial052.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-24T16:37:56.615 INFO:teuthology.orchestra.run.trial052.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 2.1 MB/s | 104 kB 00:00 2026-02-24T16:37:56.633 INFO:teuthology.orchestra.run.trial052.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 8.8 MB/s | 164 kB 00:00 2026-02-24T16:37:56.645 INFO:teuthology.orchestra.run.trial052.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 13 MB/s | 160 kB 00:00 2026-02-24T16:37:56.655 INFO:teuthology.orchestra.run.trial052.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.7 MB/s | 45 kB 00:00 2026-02-24T16:37:56.678 INFO:teuthology.orchestra.run.trial052.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 28 MB/s | 662 kB 00:00 2026-02-24T16:37:56.689 INFO:teuthology.orchestra.run.trial052.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 25 MB/s | 292 kB 00:00 2026-02-24T16:37:56.700 INFO:teuthology.orchestra.run.trial052.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 21 MB/s | 240 kB 00:00 2026-02-24T16:37:56.710 INFO:teuthology.orchestra.run.trial052.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 3.9 MB/s | 37 kB 00:00 2026-02-24T16:37:56.711 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:56.711 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:37:56.711 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:37:56.719 INFO:teuthology.orchestra.run.trial052.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 6.9 MB/s | 66 kB 00:00 2026-02-24T16:37:56.730 INFO:teuthology.orchestra.run.trial051.stdout:(35/37): ceph-common-20.3.0-5507.g618c318a.el9. 6.3 MB/s | 24 MB 00:03 2026-02-24T16:37:56.754 INFO:teuthology.orchestra.run.trial191.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:37:56.756 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:37:56.799 INFO:teuthology.orchestra.run.trial052.stdout:(26/37): pciutils-3.7.0-7.el9.x86_64.rpm 142 kB/s | 93 kB 00:00 2026-02-24T16:37:56.830 INFO:teuthology.orchestra.run.trial051.stdout:(36/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 6.7 MB/s | 2.8 MB 00:00 2026-02-24T16:37:56.869 INFO:teuthology.orchestra.run.trial052.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2026-02-24T16:37:56.889 INFO:teuthology.orchestra.run.trial052.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.2 MB/s | 25 kB 00:00 2026-02-24T16:37:56.910 INFO:teuthology.orchestra.run.trial052.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 2.3 MB/s | 49 kB 00:00 2026-02-24T16:37:56.931 INFO:teuthology.orchestra.run.trial052.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 3.1 MB/s | 67 kB 00:00 2026-02-24T16:37:56.987 INFO:teuthology.orchestra.run.trial052.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 15 MB/s | 838 kB 00:00 2026-02-24T16:37:57.010 INFO:teuthology.orchestra.run.trial052.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 8.1 MB/s | 191 kB 00:00 2026-02-24T16:37:57.048 INFO:teuthology.orchestra.run.trial052.stdout:(33/37): libarrow-9.0.0-15.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2026-02-24T16:37:57.057 INFO:teuthology.orchestra.run.trial052.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 33 MB/s | 1.6 MB 00:00 2026-02-24T16:37:57.144 INFO:teuthology.orchestra.run.trial052.stdout:(35/37): ceph-common-20.3.0-5507.g618c318a.el9. 5.5 MB/s | 24 MB 00:04 2026-02-24T16:37:57.202 INFO:teuthology.orchestra.run.trial051.stdout:(37/37): librados2-20.3.0-5507.g618c318a.el9.x8 4.2 MB/s | 3.5 MB 00:00 2026-02-24T16:37:57.206 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:37:57.206 INFO:teuthology.orchestra.run.trial051.stdout:Total 15 MB/s | 68 MB 00:04 2026-02-24T16:37:57.305 INFO:teuthology.orchestra.run.trial052.stdout:(36/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 11 MB/s | 2.8 MB 00:00 2026-02-24T16:37:57.308 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:37:57.327 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:37:57.327 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:37:57.333 INFO:teuthology.orchestra.run.trial052.stdout:(37/37): librados2-20.3.0-5507.g618c318a.el9.x8 12 MB/s | 3.5 MB 00:00 2026-02-24T16:37:57.336 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:37:57.336 INFO:teuthology.orchestra.run.trial052.stdout:Total 13 MB/s | 68 MB 00:05 2026-02-24T16:37:57.438 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:37:57.456 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:37:57.457 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:37:57.596 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:37:57.596 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:37:57.723 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:37:57.724 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:37:57.853 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:37:57.856 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:37:57.910 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:37:58.036 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:37:58.063 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:37:58.067 INFO:teuthology.orchestra.run.trial051.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:58.117 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:58.118 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:58.118 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:37:58.121 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:37:58.131 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:58.133 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:37:58.134 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:58.162 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:37:58.163 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:37:58.163 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:37:58.163 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:37:58.164 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:58.165 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:37:58.166 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:37:58.174 INFO:teuthology.orchestra.run.trial051.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:37:58.176 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:37:58.179 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:37:58.185 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:37:58.187 INFO:teuthology.orchestra.run.trial052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:37:58.191 INFO:teuthology.orchestra.run.trial052.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:58.196 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:37:58.208 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:37:58.209 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:58.232 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:58.235 INFO:teuthology.orchestra.run.trial051.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:37:58.243 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:37:58.244 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:58.256 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:37:58.259 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:37:58.259 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:58.266 INFO:teuthology.orchestra.run.trial051.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:37:58.270 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:37:58.276 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:37:58.278 INFO:teuthology.orchestra.run.trial051.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:37:58.282 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:37:58.286 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:37:58.288 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:37:58.296 INFO:teuthology.orchestra.run.trial052.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:37:58.298 INFO:teuthology.orchestra.run.trial052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:37:58.301 INFO:teuthology.orchestra.run.trial052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:37:58.305 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:37:58.316 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:37:58.327 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:37:58.328 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:58.349 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:37:58.353 INFO:teuthology.orchestra.run.trial052.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:37:58.384 INFO:teuthology.orchestra.run.trial052.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:37:58.387 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:37:58.393 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:37:58.396 INFO:teuthology.orchestra.run.trial052.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:37:58.400 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:37:58.406 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:37:58.407 INFO:teuthology.orchestra.run.trial051.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:58.436 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:58.438 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:37:58.447 INFO:teuthology.orchestra.run.trial051.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:37:58.521 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:37:58.522 INFO:teuthology.orchestra.run.trial052.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:58.551 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:37:58.553 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:37:58.561 INFO:teuthology.orchestra.run.trial052.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:37:58.732 INFO:teuthology.orchestra.run.trial051.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:37:58.733 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:58.751 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:58.755 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:37:58.761 INFO:teuthology.orchestra.run.trial051.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:37:58.763 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:37:58.764 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:58.784 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:58.847 INFO:teuthology.orchestra.run.trial052.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:37:58.848 INFO:teuthology.orchestra.run.trial052.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:58.865 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:37:58.869 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:37:58.873 INFO:teuthology.orchestra.run.trial052.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:37:58.875 INFO:teuthology.orchestra.run.trial052.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:37:58.876 INFO:teuthology.orchestra.run.trial052.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:58.894 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:37:58.897 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:37:58.899 INFO:teuthology.orchestra.run.trial051.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:59.010 INFO:teuthology.orchestra.run.trial052.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:37:59.013 INFO:teuthology.orchestra.run.trial052.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:59.168 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:59.168 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:37:59.168 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:37:59.172 INFO:teuthology.orchestra.run.trial153.stdout:lab-extras 2.1 MB/s | 50 kB 00:00 2026-02-24T16:37:59.211 INFO:teuthology.orchestra.run.trial051.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:37:59.213 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:37:59.288 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:37:59.289 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:37:59.289 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:37:59.331 INFO:teuthology.orchestra.run.trial052.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:37:59.334 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:38:00.308 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.310 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.408 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout:Upgrading: 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:38:00.410 INFO:teuthology.orchestra.run.trial153.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout:Installing weak dependencies: 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout:Install 35 Packages 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout:Upgrade 2 Packages 2026-02-24T16:38:00.411 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:00.412 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 68 M 2026-02-24T16:38:00.412 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:00.425 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.428 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.573 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.576 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:00.613 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:00.613 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:38:00.613 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:38:00.613 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:00.617 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:00.689 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:00.692 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:00.735 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:00.736 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:38:00.736 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:38:00.736 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:00.740 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:01.506 INFO:teuthology.orchestra.run.trial153.stdout:(1/37): ceph-base-20.3.0-5507.g618c318a.el9.x86 15 MB/s | 5.2 MB 00:00 2026-02-24T16:38:01.542 INFO:teuthology.orchestra.run.trial153.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 701 kB/s | 25 kB 00:00 2026-02-24T16:38:01.583 INFO:teuthology.orchestra.run.trial153.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 943 kB/s | 39 kB 00:00 2026-02-24T16:38:01.626 INFO:teuthology.orchestra.run.trial153.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 612 kB/s | 26 kB 00:00 2026-02-24T16:38:01.689 INFO:teuthology.orchestra.run.trial153.stdout:(5/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 14 MB/s | 909 kB 00:00 2026-02-24T16:38:01.741 INFO:teuthology.orchestra.run.trial153.stdout:(6/37): libradosstriper1-20.3.0-5507.g618c318a. 4.6 MB/s | 248 kB 00:00 2026-02-24T16:38:01.980 INFO:teuthology.orchestra.run.trial153.stdout:(7/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 16 MB/s | 13 MB 00:00 2026-02-24T16:38:02.011 INFO:teuthology.orchestra.run.trial153.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 1.4 MB/s | 45 kB 00:00 2026-02-24T16:38:02.052 INFO:teuthology.orchestra.run.trial153.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 4.4 MB/s | 186 kB 00:00 2026-02-24T16:38:02.093 INFO:teuthology.orchestra.run.trial153.stdout:(10/37): python3-cephfs-20.3.0-5507.g618c318a.e 4.9 MB/s | 204 kB 00:00 2026-02-24T16:38:02.125 INFO:teuthology.orchestra.run.trial153.stdout:(11/37): librgw2-20.3.0-5507.g618c318a.el9.x86_ 17 MB/s | 6.5 MB 00:00 2026-02-24T16:38:02.129 INFO:teuthology.orchestra.run.trial153.stdout:(12/37): python3-rados-20.3.0-5507.g618c318a.el 8.9 MB/s | 324 kB 00:00 2026-02-24T16:38:02.163 INFO:teuthology.orchestra.run.trial153.stdout:(13/37): ceph-common-20.3.0-5507.g618c318a.el9. 24 MB/s | 24 MB 00:01 2026-02-24T16:38:02.164 INFO:teuthology.orchestra.run.trial153.stdout:(14/37): python3-rgw-20.3.0-5507.g618c318a.el9. 2.8 MB/s | 100 kB 00:00 2026-02-24T16:38:02.165 INFO:teuthology.orchestra.run.trial153.stdout:(15/37): python3-rbd-20.3.0-5507.g618c318a.el9. 7.4 MB/s | 305 kB 00:00 2026-02-24T16:38:02.677 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:02.677 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /sys 2026-02-24T16:38:02.677 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /proc 2026-02-24T16:38:02.677 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /mnt 2026-02-24T16:38:02.678 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /var/tmp 2026-02-24T16:38:02.678 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /home 2026-02-24T16:38:02.678 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /root 2026-02-24T16:38:02.678 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /tmp 2026-02-24T16:38:02.678 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:02.735 INFO:teuthology.orchestra.run.trial153.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 58 kB/s | 33 kB 00:00 2026-02-24T16:38:02.824 INFO:teuthology.orchestra.run.trial153.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 141 kB/s | 93 kB 00:00 2026-02-24T16:38:03.114 INFO:teuthology.orchestra.run.trial153.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-24T16:38:03.143 INFO:teuthology.orchestra.run.trial153.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 255 kB/s | 104 kB 00:00 2026-02-24T16:38:03.244 INFO:teuthology.orchestra.run.trial153.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 448 kB/s | 45 kB 00:00 2026-02-24T16:38:03.259 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:03.279 INFO:teuthology.orchestra.run.trial153.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 361 kB/s | 164 kB 00:00 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:03.283 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:03.284 INFO:teuthology.orchestra.run.trial191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:03.297 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:03.297 INFO:teuthology.orchestra.run.trial191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:03.353 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:03.354 INFO:teuthology.orchestra.run.trial191.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:03.354 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:03.473 INFO:teuthology.orchestra.run.trial153.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2026-02-24T16:38:03.548 INFO:teuthology.orchestra.run.trial153.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2026-02-24T16:38:03.560 INFO:teuthology.orchestra.run.trial153.stdout:(24/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 360 kB/s | 160 kB 00:00 2026-02-24T16:38:03.587 INFO:teuthology.orchestra.run.trial153.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 2.1 MB/s | 240 kB 00:00 2026-02-24T16:38:03.627 INFO:teuthology.orchestra.run.trial153.stdout:(26/37): qatlib-service-25.08.0-2.el9.x86_64.rp 468 kB/s | 37 kB 00:00 2026-02-24T16:38:03.648 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:03.649 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:03.650 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:03.651 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:03.652 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:03.652 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:03.652 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:03.660 INFO:teuthology.orchestra.run.trial153.stdout:(27/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 663 kB/s | 66 kB 00:00 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout:Upgraded: 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.714 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:03.715 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:03.758 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-test 2026-02-24T16:38:03.842 INFO:teuthology.orchestra.run.trial153.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 136 kB/s | 25 kB 00:00 2026-02-24T16:38:03.918 INFO:teuthology.orchestra.run.trial153.stdout:(29/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 931 kB/s | 308 kB 00:00 2026-02-24T16:38:03.928 INFO:teuthology.orchestra.run.trial153.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 573 kB/s | 49 kB 00:00 2026-02-24T16:38:03.965 INFO:teuthology.orchestra.run.trial153.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2026-02-24T16:38:04.026 INFO:teuthology.orchestra.run.trial153.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 3.1 MB/s | 191 kB 00:00 2026-02-24T16:38:04.081 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:14 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:04.116 INFO:teuthology.orchestra.run.trial153.stdout:(33/37): libarrow-9.0.0-15.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2026-02-24T16:38:04.126 INFO:teuthology.orchestra.run.trial153.stdout:(34/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2026-02-24T16:38:04.166 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:04.167 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:04.167 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-24T16:38:04.167 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:04.168 INFO:teuthology.orchestra.run.trial191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:Install 10 Packages 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 80 M 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 331 M 2026-02-24T16:38:04.169 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:04.181 INFO:teuthology.orchestra.run.trial153.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2026-02-24T16:38:04.235 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:26 2026-02-24T16:38:04.272 INFO:teuthology.orchestra.run.trial153.stdout:(36/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 19 MB/s | 2.8 MB 00:00 2026-02-24T16:38:04.276 INFO:teuthology.orchestra.run.trial153.stdout:(37/37): librados2-20.3.0-5507.g618c318a.el9.x8 22 MB/s | 3.5 MB 00:00 2026-02-24T16:38:04.279 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:04.279 INFO:teuthology.orchestra.run.trial153.stdout:Total 18 MB/s | 68 MB 00:03 2026-02-24T16:38:04.380 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:04.398 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:04.398 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:04.665 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:04.666 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:04.734 INFO:teuthology.orchestra.run.trial191.stdout:(1/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 28 MB/s | 2.1 MB 00:00 2026-02-24T16:38:04.743 INFO:teuthology.orchestra.run.trial191.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 2.4 MB/s | 23 kB 00:00 2026-02-24T16:38:04.745 INFO:teuthology.orchestra.run.trial191.stdout:(3/10): libcephsqlite-20.3.0-5507.g618c318a.el9 1.8 MB/s | 159 kB 00:00 2026-02-24T16:38:04.753 INFO:teuthology.orchestra.run.trial191.stdout:(4/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 9.4 MB/s | 94 kB 00:00 2026-02-24T16:38:04.857 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:26 2026-02-24T16:38:04.976 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:05.128 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:38:05.132 INFO:teuthology.orchestra.run.trial153.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:05.182 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:05.182 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /sys 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /proc 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /mnt 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /var/tmp 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /home 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /root 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /tmp 2026-02-24T16:38:05.183 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:05.184 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:05.195 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:05.199 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:38:05.199 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:05.226 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:05.228 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:38:05.235 INFO:teuthology.orchestra.run.trial153.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:38:05.238 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:38:05.241 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:38:05.245 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:38:05.258 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:38:05.269 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:05.270 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:05.292 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:05.295 INFO:teuthology.orchestra.run.trial153.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:38:05.307 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:05.307 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /sys 2026-02-24T16:38:05.307 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /proc 2026-02-24T16:38:05.307 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /mnt 2026-02-24T16:38:05.307 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /var/tmp 2026-02-24T16:38:05.308 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /home 2026-02-24T16:38:05.308 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /root 2026-02-24T16:38:05.308 INFO:teuthology.orchestra.run.trial052.stdout:skipping the directory /tmp 2026-02-24T16:38:05.308 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:05.326 INFO:teuthology.orchestra.run.trial153.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:38:05.329 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:38:05.336 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:38:05.338 INFO:teuthology.orchestra.run.trial153.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:38:05.342 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:38:05.467 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:05.469 INFO:teuthology.orchestra.run.trial153.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:05.496 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:05.499 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:38:05.507 INFO:teuthology.orchestra.run.trial153.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:38:05.522 INFO:teuthology.orchestra.run.trial191.stdout:(5/10): libxslt-1.1.34-12.el9.x86_64.rpm 300 kB/s | 233 kB 00:00 2026-02-24T16:38:05.767 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:05.790 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:05.790 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:05.791 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:05.791 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:05.791 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:05.791 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:05.793 INFO:teuthology.orchestra.run.trial051.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:05.794 INFO:teuthology.orchestra.run.trial153.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:38:05.795 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:05.806 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:05.806 INFO:teuthology.orchestra.run.trial051.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:05.812 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:05.816 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:38:05.821 INFO:teuthology.orchestra.run.trial153.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:38:05.823 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:38:05.824 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:05.842 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:05.863 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:05.863 INFO:teuthology.orchestra.run.trial051.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:05.863 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:05.891 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:05.915 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:05.915 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:05.915 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:05.915 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:05.915 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:05.916 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:05.917 INFO:teuthology.orchestra.run.trial052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:05.919 INFO:teuthology.orchestra.run.trial191.stdout:(6/10): socat-1.7.4.1-8.el9.x86_64.rpm 764 kB/s | 303 kB 00:00 2026-02-24T16:38:05.930 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:05.930 INFO:teuthology.orchestra.run.trial052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:05.958 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:38:05.961 INFO:teuthology.orchestra.run.trial153.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:05.986 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:05.987 INFO:teuthology.orchestra.run.trial052.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:05.987 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:06.003 INFO:teuthology.orchestra.run.trial191.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 764 kB/s | 64 kB 00:00 2026-02-24T16:38:06.163 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:06.163 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:06.164 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:06.165 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:06.166 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:06.167 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:06.218 INFO:teuthology.orchestra.run.trial191.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 565 kB/s | 122 kB 00:00 2026-02-24T16:38:06.230 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:06.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:06.230 INFO:teuthology.orchestra.run.trial051.stdout:Upgraded: 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:06.231 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:06.232 INFO:teuthology.orchestra.run.trial051.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:06.233 INFO:teuthology.orchestra.run.trial051.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:06.234 INFO:teuthology.orchestra.run.trial051.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:06.234 INFO:teuthology.orchestra.run.trial051.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:06.234 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:06.234 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:06.247 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:06.248 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:38:06.248 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:06.271 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:06.271 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:06.271 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:06.271 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:06.272 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:06.273 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:06.274 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:06.275 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-test 2026-02-24T16:38:06.294 INFO:teuthology.orchestra.run.trial153.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:38:06.296 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout:Upgraded: 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.339 INFO:teuthology.orchestra.run.trial052.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:06.340 INFO:teuthology.orchestra.run.trial052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.341 INFO:teuthology.orchestra.run.trial052.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:06.342 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:06.370 INFO:teuthology.orchestra.run.trial191.stdout:(9/10): ceph-test-20.3.0-5507.g618c318a.el9.x86 44 MB/s | 76 MB 00:01 2026-02-24T16:38:06.381 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-test 2026-02-24T16:38:06.586 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:15 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:06.670 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:06.671 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:06.672 INFO:teuthology.orchestra.run.trial051.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout:Install 10 Packages 2026-02-24T16:38:06.673 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:06.674 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 80 M 2026-02-24T16:38:06.674 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 331 M 2026-02-24T16:38:06.674 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:06.699 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:16 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:06.783 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:06.784 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:06.784 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repository Size 2026-02-24T16:38:06.784 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:06.784 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout:Installing dependencies: 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:06.785 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:Install 10 Packages 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 80 M 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 331 M 2026-02-24T16:38:06.786 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:07.037 INFO:teuthology.orchestra.run.trial051.stdout:(1/10): libcephsqlite-20.3.0-5507.g618c318a.el9 1.2 MB/s | 159 kB 00:00 2026-02-24T16:38:07.386 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:07.389 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:07.495 INFO:teuthology.orchestra.run.trial191.stdout:(10/10): protobuf-3.14.0-17.el9.x86_64.rpm 375 kB/s | 1.0 MB 00:02 2026-02-24T16:38:07.500 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:07.500 INFO:teuthology.orchestra.run.trial191.stdout:Total 24 MB/s | 80 MB 00:03 2026-02-24T16:38:07.565 INFO:teuthology.orchestra.run.trial052.stdout:(1/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 48 MB/s | 2.1 MB 00:00 2026-02-24T16:38:07.567 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:07.570 INFO:teuthology.orchestra.run.trial052.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 5.1 MB/s | 23 kB 00:00 2026-02-24T16:38:07.577 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:07.577 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:07.578 INFO:teuthology.orchestra.run.trial051.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2026-02-24T16:38:07.579 INFO:teuthology.orchestra.run.trial052.stdout:(3/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 10 MB/s | 94 kB 00:00 2026-02-24T16:38:07.582 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - CRB 25 MB/s | 8.0 MB 00:00 2026-02-24T16:38:07.655 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:07.659 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:07.695 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:07.696 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:38:07.696 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:38:07.696 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:07.700 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:07.753 INFO:teuthology.orchestra.run.trial052.stdout:(4/10): libxslt-1.1.34-12.el9.x86_64.rpm 1.3 MB/s | 233 kB 00:00 2026-02-24T16:38:07.851 INFO:teuthology.orchestra.run.trial052.stdout:(5/10): protobuf-3.14.0-17.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2026-02-24T16:38:07.857 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:07.858 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:07.866 INFO:teuthology.orchestra.run.trial052.stdout:(6/10): libcephsqlite-20.3.0-5507.g618c318a.el9 462 kB/s | 159 kB 00:00 2026-02-24T16:38:07.889 INFO:teuthology.orchestra.run.trial052.stdout:(7/10): socat-1.7.4.1-8.el9.x86_64.rpm 7.8 MB/s | 303 kB 00:00 2026-02-24T16:38:07.912 INFO:teuthology.orchestra.run.trial051.stdout:(3/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-24T16:38:07.914 INFO:teuthology.orchestra.run.trial052.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.6 MB/s | 122 kB 00:00 2026-02-24T16:38:07.933 INFO:teuthology.orchestra.run.trial051.stdout:(4/10): libxslt-1.1.34-12.el9.x86_64.rpm 11 MB/s | 233 kB 00:00 2026-02-24T16:38:07.964 INFO:teuthology.orchestra.run.trial051.stdout:(5/10): protobuf-3.14.0-17.el9.x86_64.rpm 33 MB/s | 1.0 MB 00:00 2026-02-24T16:38:07.971 INFO:teuthology.orchestra.run.trial051.stdout:(6/10): socat-1.7.4.1-8.el9.x86_64.rpm 46 MB/s | 303 kB 00:00 2026-02-24T16:38:07.973 INFO:teuthology.orchestra.run.trial052.stdout:(9/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 594 kB/s | 64 kB 00:00 2026-02-24T16:38:07.975 INFO:teuthology.orchestra.run.trial051.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 14 MB/s | 64 kB 00:00 2026-02-24T16:38:08.002 INFO:teuthology.orchestra.run.trial051.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.5 MB/s | 122 kB 00:00 2026-02-24T16:38:08.160 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:08.168 INFO:teuthology.orchestra.run.trial191.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:08.176 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 8.0 MB 00:01 2026-02-24T16:38:08.178 INFO:teuthology.orchestra.run.trial191.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:08.183 INFO:teuthology.orchestra.run.trial191.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:08.187 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:08.192 INFO:teuthology.orchestra.run.trial191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:08.194 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:08.203 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:08.212 INFO:teuthology.orchestra.run.trial191.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:08.213 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:08.234 INFO:teuthology.orchestra.run.trial051.stdout:(9/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 143 kB/s | 94 kB 00:00 2026-02-24T16:38:08.678 INFO:teuthology.orchestra.run.trial051.stdout:(10/10): ceph-test-20.3.0-5507.g618c318a.el9.x8 43 MB/s | 76 MB 00:01 2026-02-24T16:38:08.680 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:08.680 INFO:teuthology.orchestra.run.trial051.stdout:Total 40 MB/s | 80 MB 00:02 2026-02-24T16:38:08.744 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:08.755 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:08.755 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:08.778 INFO:teuthology.orchestra.run.trial151.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 20 kB 00:00 2026-02-24T16:38:09.033 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:09.033 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:09.141 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 20 kB 00:00 2026-02-24T16:38:09.336 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:09.345 INFO:teuthology.orchestra.run.trial051.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:09.355 INFO:teuthology.orchestra.run.trial051.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:09.361 INFO:teuthology.orchestra.run.trial051.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:09.365 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:09.370 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:09.372 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:09.382 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:09.390 INFO:teuthology.orchestra.run.trial051.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:09.391 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:09.667 INFO:teuthology.orchestra.run.trial151.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2026-02-24T16:38:10.199 INFO:teuthology.orchestra.run.trial052.stdout:(10/10): ceph-test-20.3.0-5507.g618c318a.el9.x8 28 MB/s | 76 MB 00:02 2026-02-24T16:38:10.201 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:10.201 INFO:teuthology.orchestra.run.trial052.stdout:Total 23 MB/s | 80 MB 00:03 2026-02-24T16:38:10.268 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:10.280 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:10.280 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:10.388 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-24T16:38:10.557 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:10.557 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:10.858 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:10.868 INFO:teuthology.orchestra.run.trial052.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:10.876 INFO:teuthology.orchestra.run.trial052.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:10.881 INFO:teuthology.orchestra.run.trial052.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:10.885 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:10.890 INFO:teuthology.orchestra.run.trial052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:10.892 INFO:teuthology.orchestra.run.trial052.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:10.901 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:10.910 INFO:teuthology.orchestra.run.trial052.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:10.910 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:11.617 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:11.637 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:11.956 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:11.957 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:12.016 INFO:teuthology.orchestra.run.trial191.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:12.017 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:12.057 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /sys 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /proc 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /mnt 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /var/tmp 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /home 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /root 2026-02-24T16:38:12.244 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /tmp 2026-02-24T16:38:12.245 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:12.371 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:22 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:12.457 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:12.460 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-24T16:38:12.460 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-24T16:38:12.460 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-24T16:38:12.460 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:12.461 INFO:teuthology.orchestra.run.trial191.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:12.462 INFO:teuthology.orchestra.run.trial191.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:12.463 INFO:teuthology.orchestra.run.trial191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:12.464 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:12.465 INFO:teuthology.orchestra.run.trial191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:12.466 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:12.467 INFO:teuthology.orchestra.run.trial191.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout:Installing weak dependencies: 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:12.468 INFO:teuthology.orchestra.run.trial191.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:12.469 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:12.469 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:12.469 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-24T16:38:12.469 INFO:teuthology.orchestra.run.trial191.stdout:Install 96 Packages 2026-02-24T16:38:12.469 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:12.470 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 123 M 2026-02-24T16:38:12.470 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 656 M 2026-02-24T16:38:12.470 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:12.780 INFO:teuthology.orchestra.run.trial191.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 116 kB/s | 6.5 kB 00:00 2026-02-24T16:38:12.812 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:12.827 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:12.831 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:12.851 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:12.851 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:12.851 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:12.852 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:12.852 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:12.852 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:12.853 INFO:teuthology.orchestra.run.trial153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:12.856 INFO:teuthology.orchestra.run.trial191.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 7.1 MB/s | 950 kB 00:00 2026-02-24T16:38:12.865 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:12.866 INFO:teuthology.orchestra.run.trial153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:12.875 INFO:teuthology.orchestra.run.trial191.stdout:(3/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 16 MB/s | 2.4 MB 00:00 2026-02-24T16:38:12.880 INFO:teuthology.orchestra.run.trial191.stdout:(4/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 6.7 MB/s | 167 kB 00:00 2026-02-24T16:38:12.887 INFO:teuthology.orchestra.run.trial151.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-24T16:38:12.921 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:12.921 INFO:teuthology.orchestra.run.trial153.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:12.921 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:13.135 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:13.135 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:13.135 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:13.136 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:13.195 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:13.195 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:13.195 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:13.195 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.195 INFO:teuthology.orchestra.run.trial051.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:13.196 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:13.218 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:13.218 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:13.218 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:13.219 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:13.220 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:13.221 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:13.222 INFO:teuthology.orchestra.run.trial191.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 11 MB/s | 5.0 MB 00:00 2026-02-24T16:38:13.233 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph 2026-02-24T16:38:13.272 INFO:teuthology.orchestra.run.trial191.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 1.0 MB/s | 57 kB 00:00 2026-02-24T16:38:13.285 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:13.285 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:13.285 INFO:teuthology.orchestra.run.trial153.stdout:Upgraded: 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:13.286 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:13.287 INFO:teuthology.orchestra.run.trial153.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:13.288 INFO:teuthology.orchestra.run.trial153.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:13.289 INFO:teuthology.orchestra.run.trial153.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:13.289 INFO:teuthology.orchestra.run.trial153.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:13.289 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:13.289 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:13.331 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-test 2026-02-24T16:38:13.332 INFO:teuthology.orchestra.run.trial191.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 3.0 MB/s | 184 kB 00:00 2026-02-24T16:38:13.466 INFO:teuthology.orchestra.run.trial191.stdout:(8/96): ceph-osd-classic-20.3.0-5507.g618c318a. 26 MB/s | 16 MB 00:00 2026-02-24T16:38:13.542 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:22 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:13.572 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-24T16:38:13.628 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:13.631 INFO:teuthology.orchestra.run.trial051.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:13.632 INFO:teuthology.orchestra.run.trial051.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Installing weak dependencies: 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Install 96 Packages 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 123 M 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 656 M 2026-02-24T16:38:13.633 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:13.652 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:14 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:13.736 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:13.737 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:13.737 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:13.738 INFO:teuthology.orchestra.run.trial153.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:Install 10 Packages 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 80 M 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 331 M 2026-02-24T16:38:13.739 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:13.740 INFO:teuthology.orchestra.run.trial191.stdout:(9/96): ceph-osd-crimson-20.3.0-5507.g618c318a. 22 MB/s | 19 MB 00:00 2026-02-24T16:38:13.767 INFO:teuthology.orchestra.run.trial191.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 830 kB/s | 22 kB 00:00 2026-02-24T16:38:13.804 INFO:teuthology.orchestra.run.trial191.stdout:(11/96): ceph-mgr-diskprediction-local-20.3.0-5 22 MB/s | 7.4 MB 00:00 2026-02-24T16:38:13.805 INFO:teuthology.orchestra.run.trial191.stdout:(12/96): ceph-mgr-modules-core-20.3.0-5507.g618 7.6 MB/s | 298 kB 00:00 2026-02-24T16:38:13.818 INFO:teuthology.orchestra.run.trial191.stdout:(13/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 3.5 MB/s | 50 kB 00:00 2026-02-24T16:38:13.819 INFO:teuthology.orchestra.run.trial191.stdout:(14/96): ceph-prometheus-alerts-20.3.0-5507.g61 1.2 MB/s | 17 kB 00:00 2026-02-24T16:38:13.836 INFO:teuthology.orchestra.run.trial191.stdout:(15/96): ceph-volume-20.3.0-5507.g618c318a.el9. 16 MB/s | 301 kB 00:00 2026-02-24T16:38:13.857 INFO:teuthology.orchestra.run.trial191.stdout:(16/96): cephadm-20.3.0-5507.g618c318a.el9.noar 27 MB/s | 1.0 MB 00:00 2026-02-24T16:38:13.886 INFO:teuthology.orchestra.run.trial191.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.4 MB/s | 40 kB 00:00 2026-02-24T16:38:13.904 INFO:teuthology.orchestra.run.trial191.stdout:(18/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 5.0 MB/s | 351 kB 00:00 2026-02-24T16:38:13.906 INFO:teuthology.orchestra.run.trial191.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.6 MB/s | 72 kB 00:00 2026-02-24T16:38:13.926 INFO:teuthology.orchestra.run.trial191.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 9.1 MB/s | 184 kB 00:00 2026-02-24T16:38:13.931 INFO:teuthology.orchestra.run.trial191.stdout:(21/96): libgfortran-11.5.0-14.el9.x86_64.rpm 29 MB/s | 794 kB 00:00 2026-02-24T16:38:13.940 INFO:teuthology.orchestra.run.trial191.stdout:(22/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 17 MB/s | 253 kB 00:00 2026-02-24T16:38:13.951 INFO:teuthology.orchestra.run.trial191.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 9.6 MB/s | 106 kB 00:00 2026-02-24T16:38:13.962 INFO:teuthology.orchestra.run.trial191.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 12 MB/s | 135 kB 00:00 2026-02-24T16:38:13.967 INFO:teuthology.orchestra.run.trial191.stdout:(25/96): python3-cryptography-36.0.1-5.el9.x86_ 36 MB/s | 1.2 MB 00:00 2026-02-24T16:38:13.974 INFO:teuthology.orchestra.run.trial191.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 13 MB/s | 150 kB 00:00 2026-02-24T16:38:13.978 INFO:teuthology.orchestra.run.trial191.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 10 MB/s | 126 kB 00:00 2026-02-24T16:38:14.001 INFO:teuthology.orchestra.run.trial191.stdout:(28/96): ceph-mgr-dashboard-20.3.0-5507.g618c31 22 MB/s | 15 MB 00:00 2026-02-24T16:38:14.002 INFO:teuthology.orchestra.run.trial191.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 7.6 MB/s | 182 kB 00:00 2026-02-24T16:38:14.002 INFO:teuthology.orchestra.run.trial191.stdout:(30/96): python3-urllib3-1.26.5-7.el9.noarch.rp 7.4 MB/s | 218 kB 00:00 2026-02-24T16:38:14.034 INFO:teuthology.orchestra.run.trial191.stdout:(31/96): flexiblas-3.0.4-9.el9.x86_64.rpm 926 kB/s | 30 kB 00:00 2026-02-24T16:38:14.043 INFO:teuthology.orchestra.run.trial191.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 1.6 MB/s | 15 kB 00:00 2026-02-24T16:38:14.055 INFO:teuthology.orchestra.run.trial191.stdout:(33/96): zip-3.0-35.el9.x86_64.rpm 4.9 MB/s | 266 kB 00:00 2026-02-24T16:38:14.067 INFO:teuthology.orchestra.run.trial153.stdout:(1/10): libcephsqlite-20.3.0-5507.g618c318a.el9 1.7 MB/s | 159 kB 00:00 2026-02-24T16:38:14.075 INFO:teuthology.orchestra.run.trial191.stdout:(34/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 7.5 MB/s | 246 kB 00:00 2026-02-24T16:38:14.078 INFO:teuthology.orchestra.run.trial153.stdout:(2/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 21 MB/s | 2.1 MB 00:00 2026-02-24T16:38:14.084 INFO:teuthology.orchestra.run.trial191.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 4.6 MB/s | 42 kB 00:00 2026-02-24T16:38:14.088 INFO:teuthology.orchestra.run.trial191.stdout:(36/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 35 MB/s | 3.0 MB 00:00 2026-02-24T16:38:14.100 INFO:teuthology.orchestra.run.trial191.stdout:(37/96): lua-5.4.4-4.el9.x86_64.rpm 4.1 MB/s | 188 kB 00:00 2026-02-24T16:38:14.100 INFO:teuthology.orchestra.run.trial153.stdout:(3/10): libatomic-11.5.0-14.el9.x86_64.rpm 698 kB/s | 23 kB 00:00 2026-02-24T16:38:14.102 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repository Size 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:38:14.104 INFO:teuthology.orchestra.run.trial151.stdout:Upgrading: 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout:Installing dependencies: 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:38:14.105 INFO:teuthology.orchestra.run.trial151.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:38:14.106 INFO:teuthology.orchestra.run.trial151.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout:Installing weak dependencies: 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:38:14.107 INFO:teuthology.orchestra.run.trial151.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout:Install 35 Packages 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout:Upgrade 2 Packages 2026-02-24T16:38:14.108 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:14.109 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 68 M 2026-02-24T16:38:14.109 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:38:14.114 INFO:teuthology.orchestra.run.trial153.stdout:(4/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.5 MB/s | 94 kB 00:00 2026-02-24T16:38:14.119 INFO:teuthology.orchestra.run.trial191.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 13 MB/s | 244 kB 00:00 2026-02-24T16:38:14.130 INFO:teuthology.orchestra.run.trial191.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 21 MB/s | 249 kB 00:00 2026-02-24T16:38:14.140 INFO:teuthology.orchestra.run.trial191.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 4.8 MB/s | 48 kB 00:00 2026-02-24T16:38:14.152 INFO:teuthology.orchestra.run.trial191.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 15 MB/s | 177 kB 00:00 2026-02-24T16:38:14.167 INFO:teuthology.orchestra.run.trial191.stdout:(42/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 64 MB/s | 5.3 MB 00:00 2026-02-24T16:38:14.175 INFO:teuthology.orchestra.run.trial191.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 4.0 MB/s | 35 kB 00:00 2026-02-24T16:38:14.183 INFO:teuthology.orchestra.run.trial191.stdout:(44/96): python3-babel-2.9.1-2.el9.noarch.rpm 63 MB/s | 6.0 MB 00:00 2026-02-24T16:38:14.185 INFO:teuthology.orchestra.run.trial191.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 37 MB/s | 1.2 MB 00:00 2026-02-24T16:38:14.192 INFO:teuthology.orchestra.run.trial191.stdout:(46/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 46 MB/s | 442 kB 00:00 2026-02-24T16:38:14.195 INFO:teuthology.orchestra.run.trial191.stdout:(47/96): python3-packaging-20.9-5.el9.noarch.rp 7.3 MB/s | 77 kB 00:00 2026-02-24T16:38:14.201 INFO:teuthology.orchestra.run.trial191.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 29 MB/s | 267 kB 00:00 2026-02-24T16:38:14.206 INFO:teuthology.orchestra.run.trial191.stdout:(49/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 15 MB/s | 157 kB 00:00 2026-02-24T16:38:14.211 INFO:teuthology.orchestra.run.trial191.stdout:(50/96): python3-pyasn1-modules-0.4.8-7.el9.noa 28 MB/s | 277 kB 00:00 2026-02-24T16:38:14.216 INFO:teuthology.orchestra.run.trial191.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 5.4 MB/s | 54 kB 00:00 2026-02-24T16:38:14.225 INFO:teuthology.orchestra.run.trial191.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 4.3 MB/s | 42 kB 00:00 2026-02-24T16:38:14.268 INFO:teuthology.orchestra.run.trial153.stdout:(5/10): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-24T16:38:14.288 INFO:teuthology.orchestra.run.trial191.stdout:(53/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 55 MB/s | 6.1 MB 00:00 2026-02-24T16:38:14.289 INFO:teuthology.orchestra.run.trial191.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.9 MB/s | 189 kB 00:00 2026-02-24T16:38:14.296 INFO:teuthology.orchestra.run.trial191.stdout:(55/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 10 MB/s | 90 kB 00:00 2026-02-24T16:38:14.319 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:14.320 INFO:teuthology.orchestra.run.trial153.stdout:(6/10): protobuf-3.14.0-17.el9.x86_64.rpm 4.9 MB/s | 1.0 MB 00:00 2026-02-24T16:38:14.321 INFO:teuthology.orchestra.run.trial153.stdout:(7/10): socat-1.7.4.1-8.el9.x86_64.rpm 5.6 MB/s | 303 kB 00:00 2026-02-24T16:38:14.330 INFO:teuthology.orchestra.run.trial191.stdout:(56/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 162 MB/s | 19 MB 00:00 2026-02-24T16:38:14.337 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:14.346 INFO:teuthology.orchestra.run.trial153.stdout:(8/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-24T16:38:14.365 INFO:teuthology.orchestra.run.trial051.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 35 kB/s | 6.5 kB 00:00 2026-02-24T16:38:14.374 INFO:teuthology.orchestra.run.trial153.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.2 MB/s | 122 kB 00:00 2026-02-24T16:38:14.645 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:14.645 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:14.645 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:14.645 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:14.645 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:14.646 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:14.646 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:14.646 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:14.646 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:14.646 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:14.705 INFO:teuthology.orchestra.run.trial052.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:14.706 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:14.744 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph 2026-02-24T16:38:14.790 INFO:teuthology.orchestra.run.trial191.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.2 MB/s | 551 kB 00:00 2026-02-24T16:38:14.800 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 13 M 2026-02-24T16:38:14.802 INFO:teuthology.orchestra.run.trial183.stdout:Upgrading: 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: librados2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 3.5 M 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: librbd1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.8 M 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.2 M 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 24 M 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux x86_64 2:20.3.0-5507.g618c318a.el9 ceph 25 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 909 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T16:38:14.803 INFO:teuthology.orchestra.run.trial183.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 248 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: librgw2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 M 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 45 k 2026-02-24T16:38:14.804 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common x86_64 2:20.3.0-5507.g618c318a.el9 ceph 186 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs x86_64 2:20.3.0-5507.g618c318a.el9 ceph 204 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados x86_64 2:20.3.0-5507.g618c318a.el9 ceph 324 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 305 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw x86_64 2:20.3.0-5507.g618c318a.el9 ceph 100 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T16:38:14.805 INFO:teuthology.orchestra.run.trial183.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 39 k 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5507.g618c318a.el9 ceph 26 k 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout:Install 35 Packages 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout:Upgrade 2 Packages 2026-02-24T16:38:14.806 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:14.807 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 68 M 2026-02-24T16:38:14.807 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:38:14.825 INFO:teuthology.orchestra.run.trial191.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 565 kB/s | 19 kB 00:00 2026-02-24T16:38:14.837 INFO:teuthology.orchestra.run.trial191.stdout:(59/96): lua-devel-5.4.4-4.el9.x86_64.rpm 41 kB/s | 22 kB 00:00 2026-02-24T16:38:14.841 INFO:teuthology.orchestra.run.trial191.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 9.1 MB/s | 151 kB 00:00 2026-02-24T16:38:14.863 INFO:teuthology.orchestra.run.trial191.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.3 MB/s | 29 kB 00:00 2026-02-24T16:38:14.884 INFO:teuthology.orchestra.run.trial191.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.9 MB/s | 60 kB 00:00 2026-02-24T16:38:14.916 INFO:teuthology.orchestra.run.trial191.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 6.8 MB/s | 548 kB 00:00 2026-02-24T16:38:14.927 INFO:teuthology.orchestra.run.trial191.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2026-02-24T16:38:14.931 INFO:teuthology.orchestra.run.trial191.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 2.1 MB/s | 32 kB 00:00 2026-02-24T16:38:14.975 INFO:teuthology.orchestra.run.trial191.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 294 kB/s | 14 kB 00:00 2026-02-24T16:38:14.986 INFO:teuthology.orchestra.run.trial191.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 3.0 MB/s | 173 kB 00:00 2026-02-24T16:38:15.012 INFO:teuthology.orchestra.run.trial191.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 9.7 MB/s | 254 kB 00:00 2026-02-24T16:38:15.045 INFO:teuthology.orchestra.run.trial191.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 5.0 MB/s | 358 kB 00:00 2026-02-24T16:38:15.056 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:25 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:15.070 INFO:teuthology.orchestra.run.trial191.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 5.6 MB/s | 144 kB 00:00 2026-02-24T16:38:15.096 INFO:teuthology.orchestra.run.trial191.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.3 MB/s | 139 kB 00:00 2026-02-24T16:38:15.119 INFO:teuthology.orchestra.run.trial191.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.4 MB/s | 56 kB 00:00 2026-02-24T16:38:15.121 INFO:teuthology.orchestra.run.trial191.stdout:(73/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 19 MB/s | 2.0 MB 00:00 2026-02-24T16:38:15.142 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:15.145 INFO:teuthology.orchestra.run.trial052.stdout:======================================================================================= 2026-02-24T16:38:15.145 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repository Size 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout:======================================================================================= 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout:Installing dependencies: 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:15.146 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:15.147 INFO:teuthology.orchestra.run.trial052.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:15.148 INFO:teuthology.orchestra.run.trial052.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:15.149 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:15.150 INFO:teuthology.orchestra.run.trial052.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:15.151 INFO:teuthology.orchestra.run.trial052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:15.152 INFO:teuthology.orchestra.run.trial052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout:Installing weak dependencies: 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:15.153 INFO:teuthology.orchestra.run.trial052.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout:======================================================================================= 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout:Install 96 Packages 2026-02-24T16:38:15.154 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:15.155 INFO:teuthology.orchestra.run.trial191.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 724 kB/s | 18 kB 00:00 2026-02-24T16:38:15.155 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 123 M 2026-02-24T16:38:15.155 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 656 M 2026-02-24T16:38:15.155 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:15.158 INFO:teuthology.orchestra.run.trial191.stdout:(75/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 272 kB/s | 11 kB 00:00 2026-02-24T16:38:15.159 INFO:teuthology.orchestra.run.trial191.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 1.7 MB/s | 23 kB 00:00 2026-02-24T16:38:15.166 INFO:teuthology.orchestra.run.trial051.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 963 kB/s | 950 kB 00:00 2026-02-24T16:38:15.169 INFO:teuthology.orchestra.run.trial191.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 1.9 MB/s | 19 kB 00:00 2026-02-24T16:38:15.187 INFO:teuthology.orchestra.run.trial191.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.4 MB/s | 26 kB 00:00 2026-02-24T16:38:15.200 INFO:teuthology.orchestra.run.trial191.stdout:(79/96): python3-jaraco-context-6.0.1-3.el9.noa 471 kB/s | 20 kB 00:00 2026-02-24T16:38:15.203 INFO:teuthology.orchestra.run.trial191.stdout:(80/96): protobuf-compiler-3.14.0-17.el9.x86_64 951 kB/s | 862 kB 00:00 2026-02-24T16:38:15.239 INFO:teuthology.orchestra.run.trial191.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 2.0 MB/s | 79 kB 00:00 2026-02-24T16:38:15.255 INFO:teuthology.orchestra.run.trial191.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.6 MB/s | 86 kB 00:00 2026-02-24T16:38:15.258 INFO:teuthology.orchestra.run.trial051.stdout:(3/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 1.8 MB/s | 167 kB 00:00 2026-02-24T16:38:15.259 INFO:teuthology.orchestra.run.trial191.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.8 MB/s | 58 kB 00:00 2026-02-24T16:38:15.269 INFO:teuthology.orchestra.run.trial191.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 1.2 MB/s | 16 kB 00:00 2026-02-24T16:38:15.281 INFO:teuthology.orchestra.run.trial191.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.0 MB/s | 90 kB 00:00 2026-02-24T16:38:15.294 INFO:teuthology.orchestra.run.trial191.stdout:(86/96): python3-kubernetes-26.1.0-3.el9.noarch 9.6 MB/s | 1.0 MB 00:00 2026-02-24T16:38:15.299 INFO:teuthology.orchestra.run.trial191.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-24T16:38:15.305 INFO:teuthology.orchestra.run.trial191.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 5.6 MB/s | 59 kB 00:00 2026-02-24T16:38:15.327 INFO:teuthology.orchestra.run.trial191.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.6 MB/s | 36 kB 00:00 2026-02-24T16:38:15.342 INFO:teuthology.orchestra.run.trial191.stdout:(90/96): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2026-02-24T16:38:15.342 INFO:teuthology.orchestra.run.trial191.stdout:(91/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.9 MB/s | 125 kB 00:00 2026-02-24T16:38:15.352 INFO:teuthology.orchestra.run.trial191.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 3.4 MB/s | 86 kB 00:00 2026-02-24T16:38:15.386 INFO:teuthology.orchestra.run.trial191.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 652 kB/s | 22 kB 00:00 2026-02-24T16:38:15.392 INFO:teuthology.orchestra.run.trial191.stdout:(94/96): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2026-02-24T16:38:15.397 INFO:teuthology.orchestra.run.trial191.stdout:(95/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 889 kB/s | 48 kB 00:00 2026-02-24T16:38:15.423 INFO:teuthology.orchestra.run.trial191.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 533 kB/s | 20 kB 00:00 2026-02-24T16:38:15.425 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:15.426 INFO:teuthology.orchestra.run.trial191.stdout:Total 41 MB/s | 123 MB 00:02 2026-02-24T16:38:15.717 INFO:teuthology.orchestra.run.trial051.stdout:(4/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 1.5 MB/s | 2.4 MB 00:01 2026-02-24T16:38:15.810 INFO:teuthology.orchestra.run.trial052.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 58 kB/s | 6.5 kB 00:00 2026-02-24T16:38:15.829 INFO:teuthology.orchestra.run.trial153.stdout:(10/10): ceph-test-20.3.0-5507.g618c318a.el9.x8 41 MB/s | 76 MB 00:01 2026-02-24T16:38:15.831 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:15.831 INFO:teuthology.orchestra.run.trial153.stdout:Total 38 MB/s | 80 MB 00:02 2026-02-24T16:38:15.855 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:15.893 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:15.893 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:15.896 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:15.907 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:15.908 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:16.083 INFO:teuthology.orchestra.run.trial052.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 2.4 MB/s | 950 kB 00:00 2026-02-24T16:38:16.123 INFO:teuthology.orchestra.run.trial052.stdout:(3/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 4.0 MB/s | 167 kB 00:00 2026-02-24T16:38:16.185 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:16.185 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:16.364 INFO:teuthology.orchestra.run.trial052.stdout:(4/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 3.6 MB/s | 2.4 MB 00:00 2026-02-24T16:38:16.390 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:16.390 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:16.409 INFO:teuthology.orchestra.run.trial151.stdout:(1/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 7.2 MB/s | 13 MB 00:01 2026-02-24T16:38:16.450 INFO:teuthology.orchestra.run.trial151.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 606 kB/s | 25 kB 00:00 2026-02-24T16:38:16.487 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:16.494 INFO:teuthology.orchestra.run.trial151.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 894 kB/s | 39 kB 00:00 2026-02-24T16:38:16.496 INFO:teuthology.orchestra.run.trial153.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:16.504 INFO:teuthology.orchestra.run.trial153.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:16.509 INFO:teuthology.orchestra.run.trial153.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:16.514 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:16.518 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:16.521 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:16.530 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:16.538 INFO:teuthology.orchestra.run.trial151.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 589 kB/s | 26 kB 00:00 2026-02-24T16:38:16.539 INFO:teuthology.orchestra.run.trial153.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:16.539 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:16.560 INFO:teuthology.orchestra.run.trial051.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 2.3 MB/s | 5.0 MB 00:02 2026-02-24T16:38:16.612 INFO:teuthology.orchestra.run.trial051.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 1.1 MB/s | 57 kB 00:00 2026-02-24T16:38:16.692 INFO:teuthology.orchestra.run.trial151.stdout:(5/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 5.8 MB/s | 909 kB 00:00 2026-02-24T16:38:16.717 INFO:teuthology.orchestra.run.trial051.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 1.7 MB/s | 184 kB 00:00 2026-02-24T16:38:16.747 INFO:teuthology.orchestra.run.trial151.stdout:(6/37): libradosstriper1-20.3.0-5507.g618c318a. 4.4 MB/s | 248 kB 00:00 2026-02-24T16:38:16.900 INFO:teuthology.orchestra.run.trial183.stdout:(1/37): ceph-base-20.3.0-5507.g618c318a.el9.x86 2.7 MB/s | 5.2 MB 00:01 2026-02-24T16:38:16.917 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:16.919 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:38:16.923 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:38:16.944 INFO:teuthology.orchestra.run.trial191.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:38:16.962 INFO:teuthology.orchestra.run.trial183.stdout:(2/37): ceph-selinux-20.3.0-5507.g618c318a.el9. 408 kB/s | 25 kB 00:00 2026-02-24T16:38:16.979 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:38:16.985 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:38:16.988 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:38:16.992 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:38:17.016 INFO:teuthology.orchestra.run.trial191.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:38:17.025 INFO:teuthology.orchestra.run.trial183.stdout:(3/37): libcephfs-daemon-20.3.0-5507.g618c318a. 606 kB/s | 39 kB 00:00 2026-02-24T16:38:17.028 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:38:17.032 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:38:17.038 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:38:17.040 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:38:17.051 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:38:17.057 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:38:17.082 INFO:teuthology.orchestra.run.trial191.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:38:17.085 INFO:teuthology.orchestra.run.trial191.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:38:17.092 INFO:teuthology.orchestra.run.trial183.stdout:(4/37): libcephfs-proxy2-20.3.0-5507.g618c318a. 397 kB/s | 26 kB 00:00 2026-02-24T16:38:17.136 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:38:17.149 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:38:17.151 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:38:17.156 INFO:teuthology.orchestra.run.trial191.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:38:17.161 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:38:17.166 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:38:17.169 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:38:17.184 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:38:17.188 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:38:17.194 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:38:17.205 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:38:17.214 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:38:17.220 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:38:17.240 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:38:17.252 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:38:17.267 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:38:17.272 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:38:17.280 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:38:17.299 INFO:teuthology.orchestra.run.trial052.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 3.4 MB/s | 5.0 MB 00:01 2026-02-24T16:38:17.320 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:38:17.354 INFO:teuthology.orchestra.run.trial183.stdout:(5/37): libcephfs2-20.3.0-5507.g618c318a.el9.x8 3.4 MB/s | 909 kB 00:00 2026-02-24T16:38:17.355 INFO:teuthology.orchestra.run.trial052.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 1.0 MB/s | 57 kB 00:00 2026-02-24T16:38:17.417 INFO:teuthology.orchestra.run.trial052.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 2.9 MB/s | 184 kB 00:00 2026-02-24T16:38:17.446 INFO:teuthology.orchestra.run.trial051.stdout:(8/96): ceph-osd-classic-20.3.0-5507.g618c318a. 7.2 MB/s | 16 MB 00:02 2026-02-24T16:38:17.465 INFO:teuthology.orchestra.run.trial183.stdout:(6/37): libradosstriper1-20.3.0-5507.g618c318a. 2.2 MB/s | 248 kB 00:00 2026-02-24T16:38:17.649 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:38:17.658 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:38:17.662 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:38:17.668 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:38:17.670 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:38:17.697 INFO:teuthology.orchestra.run.trial191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:38:17.742 INFO:teuthology.orchestra.run.trial191.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:38:17.750 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:38:17.756 INFO:teuthology.orchestra.run.trial191.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:38:17.760 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:38:17.765 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:38:17.769 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:38:17.776 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:38:17.779 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:38:17.807 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:38:17.817 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:38:17.824 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:38:17.831 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:38:17.867 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:38:17.880 INFO:teuthology.orchestra.run.trial151.stdout:(7/37): librgw2-20.3.0-5507.g618c318a.el9.x86_6 5.7 MB/s | 6.5 MB 00:01 2026-02-24T16:38:17.939 INFO:teuthology.orchestra.run.trial151.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 768 kB/s | 45 kB 00:00 2026-02-24T16:38:18.001 INFO:teuthology.orchestra.run.trial151.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 2.9 MB/s | 186 kB 00:00 2026-02-24T16:38:18.065 INFO:teuthology.orchestra.run.trial151.stdout:(10/37): python3-cephfs-20.3.0-5507.g618c318a.e 3.1 MB/s | 204 kB 00:00 2026-02-24T16:38:18.093 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:38:18.117 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:38:18.121 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:38:18.130 INFO:teuthology.orchestra.run.trial151.stdout:(11/37): python3-rados-20.3.0-5507.g618c318a.el 4.8 MB/s | 324 kB 00:00 2026-02-24T16:38:18.180 INFO:teuthology.orchestra.run.trial191.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:38:18.181 INFO:teuthology.orchestra.run.trial191.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:38:18.196 INFO:teuthology.orchestra.run.trial151.stdout:(12/37): python3-rbd-20.3.0-5507.g618c318a.el9. 4.6 MB/s | 305 kB 00:00 2026-02-24T16:38:18.202 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:38:18.258 INFO:teuthology.orchestra.run.trial151.stdout:(13/37): python3-rgw-20.3.0-5507.g618c318a.el9. 1.6 MB/s | 100 kB 00:00 2026-02-24T16:38:18.276 INFO:teuthology.orchestra.run.trial051.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-55 8.9 MB/s | 7.4 MB 00:00 2026-02-24T16:38:18.306 INFO:teuthology.orchestra.run.trial151.stdout:(14/37): ceph-common-20.3.0-5507.g618c318a.el9. 6.5 MB/s | 24 MB 00:03 2026-02-24T16:38:18.323 INFO:teuthology.orchestra.run.trial051.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 473 kB/s | 22 kB 00:00 2026-02-24T16:38:18.370 INFO:teuthology.orchestra.run.trial051.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5507.g618 6.2 MB/s | 298 kB 00:00 2026-02-24T16:38:18.371 INFO:teuthology.orchestra.run.trial052.stdout:(8/96): ceph-osd-classic-20.3.0-5507.g618c318a. 7.0 MB/s | 16 MB 00:02 2026-02-24T16:38:18.410 INFO:teuthology.orchestra.run.trial051.stdout:(12/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 1.3 MB/s | 50 kB 00:00 2026-02-24T16:38:18.458 INFO:teuthology.orchestra.run.trial051.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5507.g61 364 kB/s | 17 kB 00:00 2026-02-24T16:38:18.511 INFO:teuthology.orchestra.run.trial051.stdout:(14/96): ceph-volume-20.3.0-5507.g618c318a.el9. 5.5 MB/s | 301 kB 00:00 2026-02-24T16:38:18.531 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:38:18.605 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:38:18.627 INFO:teuthology.orchestra.run.trial051.stdout:(15/96): cephadm-20.3.0-5507.g618c318a.el9.noar 8.8 MB/s | 1.0 MB 00:00 2026-02-24T16:38:18.784 INFO:teuthology.orchestra.run.trial183.stdout:(7/37): ceph-radosgw-20.3.0-5507.g618c318a.el9. 3.5 MB/s | 13 MB 00:03 2026-02-24T16:38:18.830 INFO:teuthology.orchestra.run.trial151.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 58 kB/s | 33 kB 00:00 2026-02-24T16:38:18.830 INFO:teuthology.orchestra.run.trial183.stdout:(8/37): python3-ceph-argparse-20.3.0-5507.g618c 972 kB/s | 45 kB 00:00 2026-02-24T16:38:18.880 INFO:teuthology.orchestra.run.trial183.stdout:(9/37): python3-ceph-common-20.3.0-5507.g618c31 3.7 MB/s | 186 kB 00:00 2026-02-24T16:38:18.933 INFO:teuthology.orchestra.run.trial183.stdout:(10/37): librgw2-20.3.0-5507.g618c318a.el9.x86_ 4.4 MB/s | 6.5 MB 00:01 2026-02-24T16:38:18.934 INFO:teuthology.orchestra.run.trial183.stdout:(11/37): python3-cephfs-20.3.0-5507.g618c318a.e 3.7 MB/s | 204 kB 00:00 2026-02-24T16:38:18.998 INFO:teuthology.orchestra.run.trial151.stdout:(16/37): ceph-base-20.3.0-5507.g618c318a.el9.x8 1.2 MB/s | 5.2 MB 00:04 2026-02-24T16:38:19.006 INFO:teuthology.orchestra.run.trial151.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 528 kB/s | 93 kB 00:00 2026-02-24T16:38:19.025 INFO:teuthology.orchestra.run.trial183.stdout:(12/37): python3-rados-20.3.0-5507.g618c318a.el 3.5 MB/s | 324 kB 00:00 2026-02-24T16:38:19.027 INFO:teuthology.orchestra.run.trial183.stdout:(13/37): python3-rbd-20.3.0-5507.g618c318a.el9. 3.2 MB/s | 305 kB 00:00 2026-02-24T16:38:19.068 INFO:teuthology.orchestra.run.trial183.stdout:(14/37): python3-rgw-20.3.0-5507.g618c318a.el9. 2.3 MB/s | 100 kB 00:00 2026-02-24T16:38:19.097 INFO:teuthology.orchestra.run.trial052.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-55 10 MB/s | 7.4 MB 00:00 2026-02-24T16:38:19.129 INFO:teuthology.orchestra.run.trial052.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 694 kB/s | 22 kB 00:00 2026-02-24T16:38:19.167 INFO:teuthology.orchestra.run.trial052.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5507.g618 7.8 MB/s | 298 kB 00:00 2026-02-24T16:38:19.200 INFO:teuthology.orchestra.run.trial052.stdout:(12/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 1.5 MB/s | 50 kB 00:00 2026-02-24T16:38:19.232 INFO:teuthology.orchestra.run.trial052.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5507.g61 536 kB/s | 17 kB 00:00 2026-02-24T16:38:19.274 INFO:teuthology.orchestra.run.trial052.stdout:(14/96): ceph-volume-20.3.0-5507.g618c318a.el9. 7.2 MB/s | 301 kB 00:00 2026-02-24T16:38:19.283 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:38:19.300 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:38:19.310 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:38:19.312 INFO:teuthology.orchestra.run.trial151.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 331 kB/s | 104 kB 00:00 2026-02-24T16:38:19.317 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:38:19.331 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:38:19.346 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:38:19.357 INFO:teuthology.orchestra.run.trial052.stdout:(15/96): cephadm-20.3.0-5507.g618c318a.el9.noar 12 MB/s | 1.0 MB 00:00 2026-02-24T16:38:19.422 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:38:19.431 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:38:19.455 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:38:19.468 INFO:teuthology.orchestra.run.trial183.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 75 kB/s | 33 kB 00:00 2026-02-24T16:38:19.472 INFO:teuthology.orchestra.run.trial052.stdout:(16/96): ceph-osd-crimson-20.3.0-5507.g618c318a 6.1 MB/s | 19 MB 00:03 2026-02-24T16:38:19.485 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:38:19.537 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:38:19.543 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:38:19.546 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:38:19.547 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:38:19.550 INFO:teuthology.orchestra.run.trial052.stdout:(17/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 1.8 MB/s | 351 kB 00:00 2026-02-24T16:38:19.559 INFO:teuthology.orchestra.run.trial052.stdout:(18/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 465 kB/s | 40 kB 00:00 2026-02-24T16:38:19.566 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:19.566 INFO:teuthology.orchestra.run.trial191.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:38:19.566 INFO:teuthology.orchestra.run.trial191.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:38:19.566 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:19.573 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:19.577 INFO:teuthology.orchestra.run.trial052.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 2.6 MB/s | 72 kB 00:00 2026-02-24T16:38:19.600 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:19.600 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:38:19.600 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:19.611 INFO:teuthology.orchestra.run.trial052.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 5.3 MB/s | 184 kB 00:00 2026-02-24T16:38:19.619 INFO:teuthology.orchestra.run.trial183.stdout:(16/37): ceph-common-20.3.0-5507.g618c318a.el9. 5.3 MB/s | 24 MB 00:04 2026-02-24T16:38:19.625 INFO:teuthology.orchestra.run.trial151.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 512 kB/s | 160 kB 00:00 2026-02-24T16:38:19.636 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:38:19.649 INFO:teuthology.orchestra.run.trial052.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.5 MB/s | 253 kB 00:00 2026-02-24T16:38:19.702 INFO:teuthology.orchestra.run.trial052.stdout:(22/96): ceph-mgr-dashboard-20.3.0-5507.g618c31 6.6 MB/s | 15 MB 00:02 2026-02-24T16:38:19.703 INFO:teuthology.orchestra.run.trial052.stdout:(23/96): libgfortran-11.5.0-14.el9.x86_64.rpm 5.4 MB/s | 794 kB 00:00 2026-02-24T16:38:19.727 INFO:teuthology.orchestra.run.trial051.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 319 kB/s | 351 kB 00:01 2026-02-24T16:38:19.731 INFO:teuthology.orchestra.run.trial052.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 4.8 MB/s | 135 kB 00:00 2026-02-24T16:38:19.736 INFO:teuthology.orchestra.run.trial052.stdout:(25/96): python3-cryptography-36.0.1-5.el9.x86_ 14 MB/s | 1.2 MB 00:00 2026-02-24T16:38:19.759 INFO:teuthology.orchestra.run.trial052.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 5.3 MB/s | 150 kB 00:00 2026-02-24T16:38:19.763 INFO:teuthology.orchestra.run.trial052.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 4.5 MB/s | 126 kB 00:00 2026-02-24T16:38:19.784 INFO:teuthology.orchestra.run.trial151.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2026-02-24T16:38:19.788 INFO:teuthology.orchestra.run.trial052.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 7.4 MB/s | 218 kB 00:00 2026-02-24T16:38:19.790 INFO:teuthology.orchestra.run.trial052.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 6.6 MB/s | 182 kB 00:00 2026-02-24T16:38:19.817 INFO:teuthology.orchestra.run.trial052.stdout:(30/96): zip-3.0-35.el9.x86_64.rpm 8.9 MB/s | 266 kB 00:00 2026-02-24T16:38:19.828 INFO:teuthology.orchestra.run.trial052.stdout:(31/96): python3-ply-3.11-14.el9.noarch.rpm 848 kB/s | 106 kB 00:00 2026-02-24T16:38:19.882 INFO:teuthology.orchestra.run.trial183.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 225 kB/s | 93 kB 00:00 2026-02-24T16:38:19.885 INFO:teuthology.orchestra.run.trial051.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 256 kB/s | 40 kB 00:00 2026-02-24T16:38:19.892 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:19.911 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:20.043 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libconfig-1.7.2-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libconfig-1.7.2-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.149 INFO:teuthology.orchestra.run.trial151.stdout:(21/37): nvme-cli-2.16-1.el9.x86_64.rpm 640 kB/s | 1.2 MB 00:01 2026-02-24T16:38:20.151 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.176 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.185 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.192 INFO:teuthology.orchestra.run.trial183.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 182 kB/s | 104 kB 00:00 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:20.222 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:20.223 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:20.223 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:20.223 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:20.223 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:20.257 INFO:teuthology.orchestra.run.trial191.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:20.282 INFO:teuthology.orchestra.run.trial153.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:20.283 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:20.319 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph 2026-02-24T16:38:20.371 INFO:teuthology.orchestra.run.trial051.stdout:(18/96): ceph-osd-crimson-20.3.0-5507.g618c318a 4.1 MB/s | 19 MB 00:04 2026-02-24T16:38:20.432 INFO:teuthology.orchestra.run.trial183.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 667 kB/s | 160 kB 00:00 2026-02-24T16:38:20.437 INFO:teuthology.orchestra.run.trial183.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 295 kB/s | 164 kB 00:00 2026-02-24T16:38:20.515 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.524 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.554 INFO:teuthology.orchestra.run.trial051.stdout:(19/96): ceph-mgr-dashboard-20.3.0-5507.g618c31 3.9 MB/s | 15 MB 00:03 2026-02-24T16:38:20.562 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.581 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.588 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.618 INFO:teuthology.orchestra.run.trial151.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 623 kB/s | 292 kB 00:00 2026-02-24T16:38:20.642 INFO:teuthology.orchestra.run.trial151.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 772 kB/s | 662 kB 00:00 2026-02-24T16:38:20.645 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:21 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:20.731 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:20.734 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-24T16:38:20.734 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-24T16:38:20.734 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-24T16:38:20.734 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:20.735 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:20.736 INFO:teuthology.orchestra.run.trial153.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:20.737 INFO:teuthology.orchestra.run.trial153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:20.738 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:20.739 INFO:teuthology.orchestra.run.trial153.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:20.740 INFO:teuthology.orchestra.run.trial153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:20.741 INFO:teuthology.orchestra.run.trial153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout:Installing weak dependencies: 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:20.742 INFO:teuthology.orchestra.run.trial153.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout:Install 96 Packages 2026-02-24T16:38:20.743 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:20.744 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 123 M 2026-02-24T16:38:20.744 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 656 M 2026-02-24T16:38:20.744 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:20.768 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.775 INFO:teuthology.orchestra.run.trial151.stdout:[MIRROR] qatlib-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.801 INFO:teuthology.orchestra.run.trial151.stdout:[MIRROR] qatlib-service-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.846 INFO:teuthology.orchestra.run.trial051.stdout:(20/96): libconfig-1.7.2-9.el9.x86_64.rpm 75 kB/s | 72 kB 00:00 2026-02-24T16:38:20.860 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.865 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.887 INFO:teuthology.orchestra.run.trial183.stdout:(21/37): nvme-cli-2.16-1.el9.x86_64.rpm 648 kB/s | 1.2 MB 00:01 2026-02-24T16:38:20.889 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.903 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.913 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.927 INFO:teuthology.orchestra.run.trial183.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 91 kB/s | 45 kB 00:00 2026-02-24T16:38:20.927 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.953 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libquadmath-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.970 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:20.991 INFO:teuthology.orchestra.run.trial183.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 1.2 MB/s | 662 kB 00:00 2026-02-24T16:38:21.013 INFO:teuthology.orchestra.run.trial183.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-24T16:38:21.019 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:38:21.034 INFO:teuthology.orchestra.run.trial183.stdout:(25/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2026-02-24T16:38:21.035 INFO:teuthology.orchestra.run.trial183.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.9 MB/s | 66 kB 00:00 2026-02-24T16:38:21.047 INFO:teuthology.orchestra.run.trial153.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 114 kB/s | 6.5 kB 00:00 2026-02-24T16:38:21.079 INFO:teuthology.orchestra.run.trial183.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.8 MB/s | 308 kB 00:00 2026-02-24T16:38:21.082 INFO:teuthology.orchestra.run.trial183.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 7.9 MB/s | 25 kB 00:00 2026-02-24T16:38:21.083 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:38:21.085 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:21.086 INFO:teuthology.orchestra.run.trial183.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 15 MB/s | 49 kB 00:00 2026-02-24T16:38:21.093 INFO:teuthology.orchestra.run.trial183.stdout:(30/37): libarrow-9.0.0-15.el9.x86_64.rpm 77 MB/s | 4.4 MB 00:00 2026-02-24T16:38:21.094 INFO:teuthology.orchestra.run.trial183.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 8.2 MB/s | 67 kB 00:00 2026-02-24T16:38:21.098 INFO:teuthology.orchestra.run.trial183.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 41 MB/s | 191 kB 00:00 2026-02-24T16:38:21.099 INFO:teuthology.orchestra.run.trial183.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 127 MB/s | 838 kB 00:00 2026-02-24T16:38:21.100 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:21.100 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:21.100 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:38:21.100 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:21.110 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:21.111 INFO:teuthology.orchestra.run.trial183.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 132 MB/s | 1.6 MB 00:00 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:21.134 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:21.137 INFO:teuthology.orchestra.run.trial153.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 6.3 MB/s | 950 kB 00:00 2026-02-24T16:38:21.165 INFO:teuthology.orchestra.run.trial153.stdout:(3/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 14 MB/s | 2.4 MB 00:00 2026-02-24T16:38:21.168 INFO:teuthology.orchestra.run.trial153.stdout:(4/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 5.3 MB/s | 167 kB 00:00 2026-02-24T16:38:21.193 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:21.195 INFO:teuthology.orchestra.run.trial191.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:21.205 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:38:21.229 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:38:21.231 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:21.235 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.241 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.242 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.245 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.250 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] qatlib-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.288 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.296 INFO:teuthology.orchestra.run.trial052.stdout:(32/96): flexiblas-3.0.4-9.el9.x86_64.rpm 20 kB/s | 30 kB 00:01 2026-02-24T16:38:21.299 INFO:teuthology.orchestra.run.trial052.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 10 kB/s | 15 kB 00:01 2026-02-24T16:38:21.314 INFO:teuthology.orchestra.run.trial183.stdout:(35/37): qatlib-25.08.0-2.el9.x86_64.rpm 619 kB/s | 240 kB 00:00 2026-02-24T16:38:21.315 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libquadmath-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.344 INFO:teuthology.orchestra.run.trial052.stdout:(34/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.0 MB/s | 3.0 MB 00:01 2026-02-24T16:38:21.362 INFO:teuthology.orchestra.run.trial153.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 16 MB/s | 5.0 MB 00:00 2026-02-24T16:38:21.419 INFO:teuthology.orchestra.run.trial153.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 1.0 MB/s | 57 kB 00:00 2026-02-24T16:38:21.443 INFO:teuthology.orchestra.run.trial151.stdout:[MIRROR] qatlib-service-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.481 INFO:teuthology.orchestra.run.trial153.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 2.9 MB/s | 184 kB 00:00 2026-02-24T16:38:21.588 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.634 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.640 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.686 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] openblas-0.3.29-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openblas-0.3.29-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.692 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.717 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libquadmath-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.724 INFO:teuthology.orchestra.run.trial151.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 34 kB/s | 37 kB 00:01 2026-02-24T16:38:21.732 INFO:teuthology.orchestra.run.trial183.stdout:(36/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 4.5 MB/s | 2.8 MB 00:00 2026-02-24T16:38:21.806 INFO:teuthology.orchestra.run.trial151.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 202 kB/s | 240 kB 00:01 2026-02-24T16:38:21.885 INFO:teuthology.orchestra.run.trial153.stdout:(8/96): ceph-osd-classic-20.3.0-5507.g618c318a. 22 MB/s | 16 MB 00:00 2026-02-24T16:38:21.974 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.990 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:21.998 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.013 INFO:teuthology.orchestra.run.trial183.stdout:(37/37): librados2-20.3.0-5507.g618c318a.el9.x8 3.8 MB/s | 3.5 MB 00:00 2026-02-24T16:38:22.016 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:22.016 INFO:teuthology.orchestra.run.trial183.stdout:Total 9.5 MB/s | 68 MB 00:07 2026-02-24T16:38:22.019 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.027 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] openblas-0.3.29-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openblas-0.3.29-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.057 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libquadmath-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.118 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:38:22.124 INFO:teuthology.orchestra.run.trial151.stdout:[MIRROR] qatzip-libs-1.3.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.3.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.137 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:38:22.138 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:38:22.217 INFO:teuthology.orchestra.run.trial153.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-55 22 MB/s | 7.4 MB 00:00 2026-02-24T16:38:22.265 INFO:teuthology.orchestra.run.trial153.stdout:(10/96): ceph-osd-crimson-20.3.0-5507.g618c318a 17 MB/s | 19 MB 00:01 2026-02-24T16:38:22.265 INFO:teuthology.orchestra.run.trial153.stdout:(11/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 460 kB/s | 22 kB 00:00 2026-02-24T16:38:22.282 INFO:teuthology.orchestra.run.trial153.stdout:(12/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 3.0 MB/s | 50 kB 00:00 2026-02-24T16:38:22.285 INFO:teuthology.orchestra.run.trial151.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 642 kB/s | 308 kB 00:00 2026-02-24T16:38:22.286 INFO:teuthology.orchestra.run.trial153.stdout:(13/96): ceph-mgr-modules-core-20.3.0-5507.g618 14 MB/s | 298 kB 00:00 2026-02-24T16:38:22.297 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.299 INFO:teuthology.orchestra.run.trial153.stdout:(14/96): ceph-prometheus-alerts-20.3.0-5507.g61 1.0 MB/s | 17 kB 00:00 2026-02-24T16:38:22.307 INFO:teuthology.orchestra.run.trial153.stdout:(15/96): ceph-volume-20.3.0-5507.g618c318a.el9. 14 MB/s | 301 kB 00:00 2026-02-24T16:38:22.312 INFO:teuthology.orchestra.run.trial052.stdout:(35/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 242 kB/s | 246 kB 00:01 2026-02-24T16:38:22.313 INFO:teuthology.orchestra.run.trial151.stdout:(27/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 113 kB/s | 66 kB 00:00 2026-02-24T16:38:22.320 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.328 INFO:teuthology.orchestra.run.trial052.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 183 kB/s | 188 kB 00:01 2026-02-24T16:38:22.328 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.340 INFO:teuthology.orchestra.run.trial153.stdout:(16/96): cephadm-20.3.0-5507.g618c318a.el9.noar 25 MB/s | 1.0 MB 00:00 2026-02-24T16:38:22.354 INFO:teuthology.orchestra.run.trial052.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 230 MB/s | 6.0 MB 00:00 2026-02-24T16:38:22.355 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] openblas-0.3.29-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openblas-0.3.29-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.360 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libgfortran-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.362 INFO:teuthology.orchestra.run.trial052.stdout:(38/96): openblas-0.3.29-1.el9.x86_64.rpm 41 kB/s | 42 kB 00:01 2026-02-24T16:38:22.366 INFO:teuthology.orchestra.run.trial052.stdout:(39/96): python3-devel-3.9.25-3.el9.x86_64.rpm 23 MB/s | 244 kB 00:00 2026-02-24T16:38:22.372 INFO:teuthology.orchestra.run.trial052.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 25 MB/s | 249 kB 00:00 2026-02-24T16:38:22.375 INFO:teuthology.orchestra.run.trial153.stdout:(17/96): ceph-mgr-dashboard-20.3.0-5507.g618c31 17 MB/s | 15 MB 00:00 2026-02-24T16:38:22.378 INFO:teuthology.orchestra.run.trial052.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 28 MB/s | 177 kB 00:00 2026-02-24T16:38:22.380 INFO:teuthology.orchestra.run.trial052.stdout:(42/96): python3-jmespath-1.0.1-1.el9.noarch.rp 3.2 MB/s | 48 kB 00:00 2026-02-24T16:38:22.386 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] libquadmath-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.387 INFO:teuthology.orchestra.run.trial052.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 5.3 MB/s | 35 kB 00:00 2026-02-24T16:38:22.405 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:38:22.405 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:38:22.417 INFO:teuthology.orchestra.run.trial153.stdout:(18/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.1 MB/s | 351 kB 00:00 2026-02-24T16:38:22.417 INFO:teuthology.orchestra.run.trial153.stdout:(19/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 526 kB/s | 40 kB 00:00 2026-02-24T16:38:22.426 INFO:teuthology.orchestra.run.trial052.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 156 MB/s | 6.1 MB 00:00 2026-02-24T16:38:22.440 INFO:teuthology.orchestra.run.trial153.stdout:(20/96): libconfig-1.7.2-9.el9.x86_64.rpm 1.1 MB/s | 72 kB 00:00 2026-02-24T16:38:22.440 INFO:teuthology.orchestra.run.trial052.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2026-02-24T16:38:22.443 INFO:teuthology.orchestra.run.trial052.stdout:(46/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 25 MB/s | 442 kB 00:00 2026-02-24T16:38:22.447 INFO:teuthology.orchestra.run.trial052.stdout:(47/96): python3-packaging-20.9-5.el9.noarch.rp 12 MB/s | 77 kB 00:00 2026-02-24T16:38:22.453 INFO:teuthology.orchestra.run.trial052.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 29 MB/s | 267 kB 00:00 2026-02-24T16:38:22.455 INFO:teuthology.orchestra.run.trial052.stdout:(49/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 19 MB/s | 157 kB 00:00 2026-02-24T16:38:22.461 INFO:teuthology.orchestra.run.trial052.stdout:(50/96): python3-pyasn1-modules-0.4.8-7.el9.noa 33 MB/s | 277 kB 00:00 2026-02-24T16:38:22.462 INFO:teuthology.orchestra.run.trial052.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 8.0 MB/s | 54 kB 00:00 2026-02-24T16:38:22.463 INFO:teuthology.orchestra.run.trial153.stdout:(21/96): libquadmath-11.5.0-14.el9.x86_64.rpm 4.1 MB/s | 184 kB 00:00 2026-02-24T16:38:22.470 INFO:teuthology.orchestra.run.trial052.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 4.8 MB/s | 42 kB 00:00 2026-02-24T16:38:22.477 INFO:teuthology.orchestra.run.trial052.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 27 MB/s | 189 kB 00:00 2026-02-24T16:38:22.482 INFO:teuthology.orchestra.run.trial153.stdout:(22/96): libgfortran-11.5.0-14.el9.x86_64.rpm 12 MB/s | 794 kB 00:00 2026-02-24T16:38:22.484 INFO:teuthology.orchestra.run.trial052.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 14 MB/s | 90 kB 00:00 2026-02-24T16:38:22.488 INFO:teuthology.orchestra.run.trial153.stdout:(23/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.2 MB/s | 253 kB 00:00 2026-02-24T16:38:22.501 INFO:teuthology.orchestra.run.trial052.stdout:(55/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 28 MB/s | 5.3 MB 00:00 2026-02-24T16:38:22.506 INFO:teuthology.orchestra.run.trial153.stdout:(24/96): python3-ply-3.11-14.el9.noarch.rpm 4.4 MB/s | 106 kB 00:00 2026-02-24T16:38:22.513 INFO:teuthology.orchestra.run.trial153.stdout:(25/96): python3-pycparser-2.20-6.el9.noarch.rp 5.2 MB/s | 135 kB 00:00 2026-02-24T16:38:22.530 INFO:teuthology.orchestra.run.trial153.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-24T16:38:22.531 INFO:teuthology.orchestra.run.trial151.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 114 kB/s | 25 kB 00:00 2026-02-24T16:38:22.537 INFO:teuthology.orchestra.run.trial153.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 5.2 MB/s | 126 kB 00:00 2026-02-24T16:38:22.556 INFO:teuthology.orchestra.run.trial153.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.2 MB/s | 218 kB 00:00 2026-02-24T16:38:22.561 INFO:teuthology.orchestra.run.trial153.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 7.6 MB/s | 182 kB 00:00 2026-02-24T16:38:22.568 INFO:teuthology.orchestra.run.trial153.stdout:(30/96): python3-cryptography-36.0.1-5.el9.x86_ 12 MB/s | 1.2 MB 00:00 2026-02-24T16:38:22.580 INFO:teuthology.orchestra.run.trial153.stdout:(31/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-24T16:38:22.640 INFO:teuthology.orchestra.run.trial153.stdout:(32/96): flexiblas-3.0.4-9.el9.x86_64.rpm 376 kB/s | 30 kB 00:00 2026-02-24T16:38:22.640 INFO:teuthology.orchestra.run.trial153.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 246 kB/s | 15 kB 00:00 2026-02-24T16:38:22.652 INFO:teuthology.orchestra.run.trial151.stdout:(29/37): libarrow-9.0.0-15.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2026-02-24T16:38:22.672 INFO:teuthology.orchestra.run.trial153.stdout:(34/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 7.4 MB/s | 246 kB 00:00 2026-02-24T16:38:22.677 INFO:teuthology.orchestra.run.trial153.stdout:(35/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 27 MB/s | 3.0 MB 00:00 2026-02-24T16:38:22.677 INFO:teuthology.orchestra.run.trial151.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 333 kB/s | 49 kB 00:00 2026-02-24T16:38:22.678 INFO:teuthology.orchestra.run.trial153.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 4.9 MB/s | 188 kB 00:00 2026-02-24T16:38:22.697 INFO:teuthology.orchestra.run.trial153.stdout:(37/96): openblas-0.3.29-1.el9.x86_64.rpm 1.7 MB/s | 42 kB 00:00 2026-02-24T16:38:22.719 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:38:22.723 INFO:teuthology.orchestra.run.trial151.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 953 kB/s | 67 kB 00:00 2026-02-24T16:38:22.724 INFO:teuthology.orchestra.run.trial052.stdout:(56/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 73 MB/s | 19 MB 00:00 2026-02-24T16:38:22.741 INFO:teuthology.orchestra.run.trial153.stdout:(38/96): python3-babel-2.9.1-2.el9.noarch.rpm 94 MB/s | 6.0 MB 00:00 2026-02-24T16:38:22.743 INFO:teuthology.orchestra.run.trial153.stdout:(39/96): python3-devel-3.9.25-3.el9.x86_64.rpm 5.2 MB/s | 244 kB 00:00 2026-02-24T16:38:22.749 INFO:teuthology.orchestra.run.trial153.stdout:(40/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 74 MB/s | 5.3 MB 00:00 2026-02-24T16:38:22.768 INFO:teuthology.orchestra.run.trial153.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 1.8 MB/s | 48 kB 00:00 2026-02-24T16:38:22.769 INFO:teuthology.orchestra.run.trial153.stdout:(42/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 8.8 MB/s | 249 kB 00:00 2026-02-24T16:38:22.781 INFO:teuthology.orchestra.run.trial153.stdout:(43/96): python3-libstoragemgmt-1.10.1-1.el9.x8 5.4 MB/s | 177 kB 00:00 2026-02-24T16:38:22.794 INFO:teuthology.orchestra.run.trial151.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 2.6 MB/s | 191 kB 00:00 2026-02-24T16:38:22.798 INFO:teuthology.orchestra.run.trial153.stdout:(44/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 41 MB/s | 1.2 MB 00:00 2026-02-24T16:38:22.799 INFO:teuthology.orchestra.run.trial153.stdout:(45/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.2 MB/s | 35 kB 00:00 2026-02-24T16:38:22.823 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:22.824 INFO:teuthology.orchestra.run.trial153.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.0 MB/s | 77 kB 00:00 2026-02-24T16:38:22.829 INFO:teuthology.orchestra.run.trial153.stdout:(47/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 14 MB/s | 442 kB 00:00 2026-02-24T16:38:22.830 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:22.854 INFO:teuthology.orchestra.run.trial153.stdout:(48/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 84 MB/s | 6.1 MB 00:00 2026-02-24T16:38:22.855 INFO:teuthology.orchestra.run.trial153.stdout:(49/96): python3-protobuf-3.14.0-17.el9.noarch. 8.5 MB/s | 267 kB 00:00 2026-02-24T16:38:22.855 INFO:teuthology.orchestra.run.trial153.stdout:(50/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 5.9 MB/s | 157 kB 00:00 2026-02-24T16:38:22.871 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:38:22.876 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:22.878 INFO:teuthology.orchestra.run.trial153.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 2.3 MB/s | 54 kB 00:00 2026-02-24T16:38:22.881 INFO:teuthology.orchestra.run.trial153.stdout:(52/96): python3-pyasn1-modules-0.4.8-7.el9.noa 10 MB/s | 277 kB 00:00 2026-02-24T16:38:22.896 INFO:teuthology.orchestra.run.trial151.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2026-02-24T16:38:22.906 INFO:teuthology.orchestra.run.trial153.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 42 kB 00:00 2026-02-24T16:38:22.908 INFO:teuthology.orchestra.run.trial153.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 6.9 MB/s | 189 kB 00:00 2026-02-24T16:38:22.928 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:22.929 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:22.931 INFO:teuthology.orchestra.run.trial153.stdout:(55/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.5 MB/s | 90 kB 00:00 2026-02-24T16:38:22.943 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:22.946 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:38:22.947 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:22.949 INFO:teuthology.orchestra.run.trial153.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 551 kB/s | 22 kB 00:00 2026-02-24T16:38:22.964 INFO:teuthology.orchestra.run.trial153.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 26 MB/s | 862 kB 00:00 2026-02-24T16:38:22.965 INFO:teuthology.orchestra.run.trial052.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.2 MB/s | 551 kB 00:00 2026-02-24T16:38:22.971 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.974 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:22.978 INFO:teuthology.orchestra.run.trial052.stdout:[MIRROR] protobuf-compiler-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:22.979 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:38:22.987 INFO:teuthology.orchestra.run.trial051.stdout:(21/96): libgfortran-11.5.0-14.el9.x86_64.rpm 304 kB/s | 794 kB 00:02 2026-02-24T16:38:22.988 INFO:teuthology.orchestra.run.trial183.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:38:22.990 INFO:teuthology.orchestra.run.trial153.stdout:(58/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 143 MB/s | 19 MB 00:00 2026-02-24T16:38:22.991 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:38:22.992 INFO:teuthology.orchestra.run.trial052.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 716 kB/s | 19 kB 00:00 2026-02-24T16:38:22.995 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:38:22.999 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:38:23.009 INFO:teuthology.orchestra.run.trial151.stdout:(34/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2026-02-24T16:38:23.011 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:38:23.018 INFO:teuthology.orchestra.run.trial051.stdout:(22/96): libquadmath-11.5.0-14.el9.x86_64.rpm 75 kB/s | 184 kB 00:02 2026-02-24T16:38:23.022 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:23.023 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:23.024 INFO:teuthology.orchestra.run.trial052.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 4.7 MB/s | 151 kB 00:00 2026-02-24T16:38:23.047 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:23.050 INFO:teuthology.orchestra.run.trial183.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:38:23.068 INFO:teuthology.orchestra.run.trial052.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 12 MB/s | 548 kB 00:00 2026-02-24T16:38:23.082 INFO:teuthology.orchestra.run.trial183.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:38:23.087 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:38:23.093 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:38:23.094 INFO:teuthology.orchestra.run.trial052.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.1 MB/s | 29 kB 00:00 2026-02-24T16:38:23.095 INFO:teuthology.orchestra.run.trial183.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:38:23.099 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:38:23.122 INFO:teuthology.orchestra.run.trial153.stdout:(59/96): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2026-02-24T16:38:23.123 INFO:teuthology.orchestra.run.trial052.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.1 MB/s | 60 kB 00:00 2026-02-24T16:38:23.147 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cryptography-36.0.1-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:23.150 INFO:teuthology.orchestra.run.trial052.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.6 MB/s | 43 kB 00:00 2026-02-24T16:38:23.159 INFO:teuthology.orchestra.run.trial151.stdout:(35/37): librbd1-20.3.0-5507.g618c318a.el9.x86_ 19 MB/s | 2.8 MB 00:00 2026-02-24T16:38:23.178 INFO:teuthology.orchestra.run.trial052.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.2 MB/s | 32 kB 00:00 2026-02-24T16:38:23.204 INFO:teuthology.orchestra.run.trial052.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 524 kB/s | 14 kB 00:00 2026-02-24T16:38:23.220 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:23.222 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:23.236 INFO:teuthology.orchestra.run.trial052.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.3 MB/s | 173 kB 00:00 2026-02-24T16:38:23.248 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:23.250 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:38:23.256 INFO:teuthology.orchestra.run.trial183.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:38:23.272 INFO:teuthology.orchestra.run.trial052.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 9.8 MB/s | 358 kB 00:00 2026-02-24T16:38:23.285 INFO:teuthology.orchestra.run.trial153.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 513 kB/s | 151 kB 00:00 2026-02-24T16:38:23.286 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:23.287 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:23.299 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:23.300 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:23.306 INFO:teuthology.orchestra.run.trial052.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 7.4 MB/s | 254 kB 00:00 2026-02-24T16:38:23.345 INFO:teuthology.orchestra.run.trial153.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 497 kB/s | 29 kB 00:00 2026-02-24T16:38:23.358 INFO:teuthology.orchestra.run.trial153.stdout:(62/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.3 MB/s | 551 kB 00:00 2026-02-24T16:38:23.361 INFO:teuthology.orchestra.run.trial151.stdout:(36/37): librados2-20.3.0-5507.g618c318a.el9.x8 7.5 MB/s | 3.5 MB 00:00 2026-02-24T16:38:23.362 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:23.387 INFO:teuthology.orchestra.run.trial052.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 25 MB/s | 2.0 MB 00:00 2026-02-24T16:38:23.404 INFO:teuthology.orchestra.run.trial153.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 1.9 MB/s | 548 kB 00:00 2026-02-24T16:38:23.405 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:38:23.406 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:23.412 INFO:teuthology.orchestra.run.trial153.stdout:(64/96): python3-backports-tarfile-1.2.0-1.el9. 899 kB/s | 60 kB 00:00 2026-02-24T16:38:23.413 INFO:teuthology.orchestra.run.trial153.stdout:(65/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 795 kB/s | 43 kB 00:00 2026-02-24T16:38:23.416 INFO:teuthology.orchestra.run.trial052.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 5.0 MB/s | 144 kB 00:00 2026-02-24T16:38:23.428 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:23.428 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:23.428 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:38:23.429 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:23.429 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:23.429 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:23.440 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:23.443 INFO:teuthology.orchestra.run.trial052.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.9 MB/s | 139 kB 00:00 2026-02-24T16:38:23.458 INFO:teuthology.orchestra.run.trial153.stdout:(66/96): python3-cachetools-4.2.4-1.el9.noarch. 595 kB/s | 32 kB 00:00 2026-02-24T16:38:23.467 INFO:teuthology.orchestra.run.trial153.stdout:(67/96): python3-certifi-2023.05.07-4.el9.noarc 253 kB/s | 14 kB 00:00 2026-02-24T16:38:23.470 INFO:teuthology.orchestra.run.trial052.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.0 MB/s | 56 kB 00:00 2026-02-24T16:38:23.476 INFO:teuthology.orchestra.run.trial153.stdout:(68/96): python3-cheroot-10.0.1-4.el9.noarch.rp 2.7 MB/s | 173 kB 00:00 2026-02-24T16:38:23.485 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-cryptography-36.0.1-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T16:38:23.493 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:23.497 INFO:teuthology.orchestra.run.trial052.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 402 kB/s | 11 kB 00:00 2026-02-24T16:38:23.518 INFO:teuthology.orchestra.run.trial051.stdout:(23/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 95 kB/s | 253 kB 00:02 2026-02-24T16:38:23.524 INFO:teuthology.orchestra.run.trial052.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 666 kB/s | 18 kB 00:00 2026-02-24T16:38:23.533 INFO:teuthology.orchestra.run.trial153.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2026-02-24T16:38:23.542 INFO:teuthology.orchestra.run.trial183.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:38:23.544 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:23.550 INFO:teuthology.orchestra.run.trial052.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 871 kB/s | 23 kB 00:00 2026-02-24T16:38:23.560 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:23.564 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:38:23.568 INFO:teuthology.orchestra.run.trial183.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:38:23.571 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:38:23.571 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:23.578 INFO:teuthology.orchestra.run.trial052.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 733 kB/s | 20 kB 00:00 2026-02-24T16:38:23.582 INFO:teuthology.orchestra.run.trial153.stdout:(70/96): python3-google-auth-2.45.0-1.el9.noarc 2.2 MB/s | 254 kB 00:00 2026-02-24T16:38:23.588 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:23.595 INFO:teuthology.orchestra.run.trial153.stdout:(71/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.3 MB/s | 144 kB 00:00 2026-02-24T16:38:23.604 INFO:teuthology.orchestra.run.trial052.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 725 kB/s | 19 kB 00:00 2026-02-24T16:38:23.631 INFO:teuthology.orchestra.run.trial052.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 987 kB/s | 26 kB 00:00 2026-02-24T16:38:23.643 INFO:teuthology.orchestra.run.trial153.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2026-02-24T16:38:23.650 INFO:teuthology.orchestra.run.trial153.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.0 MB/s | 56 kB 00:00 2026-02-24T16:38:23.652 INFO:teuthology.orchestra.run.trial153.stdout:(74/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.0 MB/s | 139 kB 00:00 2026-02-24T16:38:23.663 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:23.665 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:23.667 INFO:teuthology.orchestra.run.trial052.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 28 MB/s | 1.0 MB 00:00 2026-02-24T16:38:23.686 INFO:teuthology.orchestra.run.trial052.stdout:(80/96): lua-devel-5.4.4-4.el9.x86_64.rpm 19 kB/s | 22 kB 00:01 2026-02-24T16:38:23.687 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:23.687 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:23.687 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:38:23.687 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:23.687 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:23.688 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:23.696 INFO:teuthology.orchestra.run.trial052.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 2.8 MB/s | 79 kB 00:00 2026-02-24T16:38:23.696 INFO:teuthology.orchestra.run.trial153.stdout:(75/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 201 kB/s | 11 kB 00:00 2026-02-24T16:38:23.702 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:38:23.703 INFO:teuthology.orchestra.run.trial153.stdout:(76/96): python3-jaraco-classes-3.2.1-5.el9.noa 333 kB/s | 18 kB 00:00 2026-02-24T16:38:23.704 INFO:teuthology.orchestra.run.trial183.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:23.706 INFO:teuthology.orchestra.run.trial153.stdout:(77/96): python3-jaraco-collections-3.0.0-8.el9 429 kB/s | 23 kB 00:00 2026-02-24T16:38:23.723 INFO:teuthology.orchestra.run.trial052.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.1 MB/s | 58 kB 00:00 2026-02-24T16:38:23.750 INFO:teuthology.orchestra.run.trial052.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 613 kB/s | 16 kB 00:00 2026-02-24T16:38:23.750 INFO:teuthology.orchestra.run.trial153.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 368 kB/s | 20 kB 00:00 2026-02-24T16:38:23.757 INFO:teuthology.orchestra.run.trial153.stdout:(79/96): python3-jaraco-functools-3.5.0-2.el9.n 365 kB/s | 19 kB 00:00 2026-02-24T16:38:23.760 INFO:teuthology.orchestra.run.trial153.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 490 kB/s | 26 kB 00:00 2026-02-24T16:38:23.777 INFO:teuthology.orchestra.run.trial052.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.2 MB/s | 90 kB 00:00 2026-02-24T16:38:23.782 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:23.803 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:23.803 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:23.803 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:38:23.803 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:23.803 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:23.804 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:23.805 INFO:teuthology.orchestra.run.trial052.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.1 MB/s | 31 kB 00:00 2026-02-24T16:38:23.805 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:23.811 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:23.814 INFO:teuthology.orchestra.run.trial153.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 1.4 MB/s | 79 kB 00:00 2026-02-24T16:38:23.820 INFO:teuthology.orchestra.run.trial153.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.4 MB/s | 86 kB 00:00 2026-02-24T16:38:23.824 INFO:teuthology.orchestra.run.trial153.stdout:(83/96): python3-kubernetes-26.1.0-3.el9.noarch 14 MB/s | 1.0 MB 00:00 2026-02-24T16:38:23.832 INFO:teuthology.orchestra.run.trial052.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 6.5 MB/s | 188 kB 00:00 2026-02-24T16:38:23.837 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:23.840 INFO:teuthology.orchestra.run.trial052.stdout:(87/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 562 kB/s | 86 kB 00:00 2026-02-24T16:38:23.859 INFO:teuthology.orchestra.run.trial052.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 2.1 MB/s | 59 kB 00:00 2026-02-24T16:38:23.870 INFO:teuthology.orchestra.run.trial153.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.0 MB/s | 58 kB 00:00 2026-02-24T16:38:23.873 INFO:teuthology.orchestra.run.trial153.stdout:(85/96): python3-portend-3.1.0-2.el9.noarch.rpm 309 kB/s | 16 kB 00:00 2026-02-24T16:38:23.878 INFO:teuthology.orchestra.run.trial153.stdout:(86/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.6 MB/s | 90 kB 00:00 2026-02-24T16:38:23.886 INFO:teuthology.orchestra.run.trial052.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.3 MB/s | 36 kB 00:00 2026-02-24T16:38:23.896 INFO:teuthology.orchestra.run.trial052.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.2 MB/s | 125 kB 00:00 2026-02-24T16:38:23.913 INFO:teuthology.orchestra.run.trial052.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 3.1 MB/s | 86 kB 00:00 2026-02-24T16:38:23.918 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pycparser-2.20-6.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pycparser-2.20-6.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:23.924 INFO:teuthology.orchestra.run.trial153.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 567 kB/s | 31 kB 00:00 2026-02-24T16:38:23.929 INFO:teuthology.orchestra.run.trial052.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2026-02-24T16:38:23.932 INFO:teuthology.orchestra.run.trial153.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2026-02-24T16:38:23.940 INFO:teuthology.orchestra.run.trial052.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.8 MB/s | 48 kB 00:00 2026-02-24T16:38:23.941 INFO:teuthology.orchestra.run.trial153.stdout:(89/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2026-02-24T16:38:23.956 INFO:teuthology.orchestra.run.trial052.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 826 kB/s | 22 kB 00:00 2026-02-24T16:38:23.966 INFO:teuthology.orchestra.run.trial052.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 756 kB/s | 20 kB 00:00 2026-02-24T16:38:23.985 INFO:teuthology.orchestra.run.trial153.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 672 kB/s | 36 kB 00:00 2026-02-24T16:38:23.985 INFO:teuthology.orchestra.run.trial153.stdout:(91/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.0 MB/s | 125 kB 00:00 2026-02-24T16:38:23.991 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:23.991 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:38:23.992 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:24.001 INFO:teuthology.orchestra.run.trial153.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 1.4 MB/s | 86 kB 00:00 2026-02-24T16:38:24.039 INFO:teuthology.orchestra.run.trial183.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:38:24.040 INFO:teuthology.orchestra.run.trial153.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 1.6 MB/s | 90 kB 00:00 2026-02-24T16:38:24.041 INFO:teuthology.orchestra.run.trial153.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 875 kB/s | 48 kB 00:00 2026-02-24T16:38:24.041 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:38:24.055 INFO:teuthology.orchestra.run.trial153.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 409 kB/s | 22 kB 00:00 2026-02-24T16:38:24.094 INFO:teuthology.orchestra.run.trial153.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 371 kB/s | 20 kB 00:00 2026-02-24T16:38:24.097 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:24.097 INFO:teuthology.orchestra.run.trial153.stdout:Total 37 MB/s | 123 MB 00:03 2026-02-24T16:38:24.143 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:24.315 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pycparser-2.20-6.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pycparser-2.20-6.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:24.465 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:24.512 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:24.550 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:24.551 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:38:24.620 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:38:24.621 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:38:24.622 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:38:24.623 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:38:24.637 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pycparser-2.20-6.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pycparser-2.20-6.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:24.693 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:38:24.693 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.694 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:38:24.695 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:38:24.696 INFO:teuthology.orchestra.run.trial191.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:38:24.697 INFO:teuthology.orchestra.run.trial191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:38:24.698 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:24.699 INFO:teuthology.orchestra.run.trial191.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:38:24.700 INFO:teuthology.orchestra.run.trial191.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:38:24.701 INFO:teuthology.orchestra.run.trial191.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:38:24.702 INFO:teuthology.orchestra.run.trial191.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:38:24.702 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:24.702 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:24.702 INFO:teuthology.orchestra.run.trial052.stdout:(96/96): protobuf-compiler-3.14.0-17.el9.x86_64 392 kB/s | 862 kB 00:02 2026-02-24T16:38:24.707 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:24.707 INFO:teuthology.orchestra.run.trial052.stdout:Total 13 MB/s | 123 MB 00:09 2026-02-24T16:38:24.750 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-base 2026-02-24T16:38:24.979 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pycparser-2.20-6.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pycparser-2.20-6.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.038 INFO:teuthology.orchestra.run.trial051.stdout:(24/96): python3-ply-3.11-14.el9.noarch.rpm 53 kB/s | 106 kB 00:02 2026-02-24T16:38:25.045 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:25.045 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:25.071 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:35 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:25.136 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:25.138 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:25.144 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:25.147 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:25.155 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:25.155 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:25.155 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:25.178 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install cephadm 2026-02-24T16:38:25.179 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:25.180 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:25.196 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pyparsing-2.4.7-9.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pyparsing-2.4.7-9.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.301 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pycparser-2.20-6.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pycparser-2.20-6.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.388 INFO:teuthology.orchestra.run.trial051.stdout:(25/96): python3-pycparser-2.20-6.el9.noarch.rp 72 kB/s | 135 kB 00:01 2026-02-24T16:38:25.408 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:25.412 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:25.450 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:25.450 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:38:25.450 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:38:25.450 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:25.451 INFO:teuthology.orchestra.run.trial051.stdout:(26/96): python3-cryptography-36.0.1-5.el9.x86_ 518 kB/s | 1.2 MB 00:02 2026-02-24T16:38:25.453 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:25.495 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:35 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:25.521 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pyparsing-2.4.7-9.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pyparsing-2.4.7-9.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.547 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-requests-2.25.1-10.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-requests-2.25.1-10.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.564 INFO:teuthology.orchestra.run.trial191.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:25.569 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:25.571 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:38:25.575 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:38:25.583 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:25.584 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:25.584 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:25.596 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:38:25.606 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:38:25.631 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:38:25.637 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:38:25.640 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:38:25.644 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:38:25.667 INFO:teuthology.orchestra.run.trial153.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:38:25.680 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:38:25.681 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:25.681 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:25.683 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:38:25.689 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:38:25.691 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:38:25.703 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:38:25.708 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:38:25.733 INFO:teuthology.orchestra.run.trial153.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:38:25.737 INFO:teuthology.orchestra.run.trial153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:38:25.788 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:38:25.801 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:38:25.803 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:38:25.808 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:38:25.813 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:38:25.818 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:38:25.821 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:38:25.836 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:38:25.840 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:38:25.846 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:38:25.853 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-urllib3-1.26.5-7.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-urllib3-1.26.5-7.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.857 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:38:25.866 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:38:25.875 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:38:25.895 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:38:25.907 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:38:25.914 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:35 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:25.921 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pyparsing-2.4.7-9.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pyparsing-2.4.7-9.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.922 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:38:25.927 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:38:25.934 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:38:25.945 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-requests-2.25.1-10.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-requests-2.25.1-10.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:25.975 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:38:25.998 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:25.999 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:25.999 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-24T16:38:25.999 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:25.999 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 147 k 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 419 k 2026-02-24T16:38:26.000 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:26.214 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:26.216 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:38:26.219 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:38:26.241 INFO:teuthology.orchestra.run.trial052.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:38:26.248 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-urllib3-1.26.5-7.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-urllib3-1.26.5-7.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:26.259 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-pyparsing-2.4.7-9.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-pyparsing-2.4.7-9.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:26.271 INFO:teuthology.orchestra.run.trial051.stdout:[MIRROR] python3-requests-2.25.1-10.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-requests-2.25.1-10.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-24T16:38:26.275 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:38:26.281 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:38:26.283 INFO:teuthology.orchestra.run.trial052.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:38:26.288 INFO:teuthology.orchestra.run.trial052.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:38:26.304 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:38:26.311 INFO:teuthology.orchestra.run.trial052.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:38:26.313 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:38:26.317 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:38:26.323 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:38:26.324 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:38:26.325 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:38:26.327 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:38:26.333 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:38:26.335 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:38:26.346 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:38:26.346 INFO:teuthology.orchestra.run.trial051.stdout:(27/96): python3-pyparsing-2.4.7-9.el9.noarch.r 115 kB/s | 150 kB 00:01 2026-02-24T16:38:26.352 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:38:26.355 INFO:teuthology.orchestra.run.trial153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:38:26.357 INFO:teuthology.orchestra.run.trial191.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 411 kB/s | 147 kB 00:00 2026-02-24T16:38:26.357 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:26.357 INFO:teuthology.orchestra.run.trial191.stdout:Total 410 kB/s | 147 kB 00:00 2026-02-24T16:38:26.357 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:26.363 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:26.363 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:26.376 INFO:teuthology.orchestra.run.trial052.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:38:26.379 INFO:teuthology.orchestra.run.trial052.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:38:26.394 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:26.394 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:26.400 INFO:teuthology.orchestra.run.trial153.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:38:26.409 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:38:26.414 INFO:teuthology.orchestra.run.trial151.stdout:(37/37): libnbd-1.20.3-4.el9.x86_64.rpm 22 kB/s | 164 kB 00:07 2026-02-24T16:38:26.415 INFO:teuthology.orchestra.run.trial153.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:38:26.419 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:38:26.420 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:26.420 INFO:teuthology.orchestra.run.trial151.stdout:Total 5.6 MB/s | 68 MB 00:12 2026-02-24T16:38:26.425 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:38:26.428 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:38:26.429 INFO:teuthology.orchestra.run.trial052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:38:26.435 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:38:26.439 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:38:26.442 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:38:26.444 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:38:26.449 INFO:teuthology.orchestra.run.trial052.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:38:26.450 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:26.450 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:26.454 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:38:26.459 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:38:26.462 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:38:26.467 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:38:26.473 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:26.473 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:26.473 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:38:26.473 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:26.476 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:38:26.476 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:38:26.481 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:38:26.483 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:38:26.486 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:38:26.490 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:38:26.497 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:38:26.506 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:38:26.513 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:38:26.515 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:38:26.524 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:38:26.532 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:38:26.534 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:38:26.534 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:38:26.543 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:38:26.559 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:38:26.563 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:38:26.571 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:38:26.611 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:38:26.687 INFO:teuthology.orchestra.run.trial051.stdout:(28/96): python3-requests-2.25.1-10.el9.noarch. 97 kB/s | 126 kB 00:01 2026-02-24T16:38:26.751 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:38:26.776 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:38:26.780 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:38:26.794 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:26.794 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:26.794 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:26.794 INFO:teuthology.orchestra.run.trial191.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:26.794 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:26.795 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:26.800 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:38:26.800 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:38:26.824 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr 2026-02-24T16:38:26.839 INFO:teuthology.orchestra.run.trial153.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:38:26.841 INFO:teuthology.orchestra.run.trial153.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:38:26.863 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:38:26.903 INFO:teuthology.orchestra.run.trial051.stdout:(29/96): python3-urllib3-1.26.5-7.el9.noarch.rp 150 kB/s | 218 kB 00:01 2026-02-24T16:38:26.936 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:38:26.946 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:38:26.950 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:38:26.956 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:38:26.957 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:38:26.985 INFO:teuthology.orchestra.run.trial052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:38:26.989 INFO:teuthology.orchestra.run.trial051.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 345 kB/s | 30 kB 00:00 2026-02-24T16:38:27.029 INFO:teuthology.orchestra.run.trial052.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:38:27.037 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:38:27.043 INFO:teuthology.orchestra.run.trial052.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:38:27.047 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:38:27.052 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:38:27.056 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:38:27.058 INFO:teuthology.orchestra.run.trial051.stdout:(31/96): unzip-6.0-59.el9.x86_64.rpm 255 kB/s | 182 kB 00:00 2026-02-24T16:38:27.063 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:38:27.066 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:38:27.094 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:38:27.103 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:38:27.110 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:38:27.113 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:38:27.116 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:38:27.141 INFO:teuthology.orchestra.run.trial051.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 178 kB/s | 15 kB 00:00 2026-02-24T16:38:27.142 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:37 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:27.151 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:38:27.190 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:38:27.207 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:27.226 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:27.227 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:27.227 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:27.248 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:38:27.263 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:38:27.266 INFO:teuthology.orchestra.run.trial151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-24T16:38:27.269 INFO:teuthology.orchestra.run.trial151.stdout: Upgrading : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:27.319 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:27.320 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:27.332 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:27.335 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 4/39 2026-02-24T16:38:27.335 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:27.363 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:27.365 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 6/39 2026-02-24T16:38:27.373 INFO:teuthology.orchestra.run.trial151.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-24T16:38:27.375 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:38:27.375 INFO:teuthology.orchestra.run.trial151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-24T16:38:27.378 INFO:teuthology.orchestra.run.trial151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-24T16:38:27.382 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-24T16:38:27.394 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 11/39 2026-02-24T16:38:27.398 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:38:27.402 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:38:27.405 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:27.406 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:27.427 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 13/39 2026-02-24T16:38:27.431 INFO:teuthology.orchestra.run.trial151.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-24T16:38:27.460 INFO:teuthology.orchestra.run.trial051.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 775 kB/s | 246 kB 00:00 2026-02-24T16:38:27.462 INFO:teuthology.orchestra.run.trial052.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:38:27.462 INFO:teuthology.orchestra.run.trial151.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-24T16:38:27.463 INFO:teuthology.orchestra.run.trial052.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:38:27.466 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-24T16:38:27.472 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-24T16:38:27.475 INFO:teuthology.orchestra.run.trial151.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-24T16:38:27.479 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-24T16:38:27.485 INFO:teuthology.orchestra.run.trial052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:38:27.499 INFO:teuthology.orchestra.run.trial051.stdout:(34/96): zip-3.0-35.el9.x86_64.rpm 328 kB/s | 266 kB 00:00 2026-02-24T16:38:27.562 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:37 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:27.583 INFO:teuthology.orchestra.run.trial051.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 498 kB/s | 42 kB 00:00 2026-02-24T16:38:27.601 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:27.602 INFO:teuthology.orchestra.run.trial151.stdout: Upgrading : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:27.627 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:27.627 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 21/39 2026-02-24T16:38:27.629 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 22/39 2026-02-24T16:38:27.635 INFO:teuthology.orchestra.run.trial151.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-24T16:38:27.645 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:27.646 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:27.646 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:27.668 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:38:27.701 INFO:teuthology.orchestra.run.trial051.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 783 kB/s | 188 kB 00:00 2026-02-24T16:38:27.810 INFO:teuthology.orchestra.run.trial052.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:38:27.884 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:38:27.919 INFO:teuthology.orchestra.run.trial151.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-24T16:38:27.920 INFO:teuthology.orchestra.run.trial151.stdout: Installing : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:27.936 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 25/39 2026-02-24T16:38:27.938 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:38:27.940 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 26/39 2026-02-24T16:38:27.944 INFO:teuthology.orchestra.run.trial151.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-24T16:38:27.947 INFO:teuthology.orchestra.run.trial151.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-24T16:38:27.947 INFO:teuthology.orchestra.run.trial151.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:27.956 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:38:27.964 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-24T16:38:27.966 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:38:27.972 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:38:27.984 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:37 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:27.986 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:38:28.001 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:38:28.049 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:28.067 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:28.068 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:28.068 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:28.076 INFO:teuthology.orchestra.run.trial151.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-24T16:38:28.077 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:38:28.079 INFO:teuthology.orchestra.run.trial151.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:28.086 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:38:28.090 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:38:28.110 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:38:28.140 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:38:28.192 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:38:28.198 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:38:28.201 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:38:28.203 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:38:28.221 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.221 INFO:teuthology.orchestra.run.trial153.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:38:28.221 INFO:teuthology.orchestra.run.trial153.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:38:28.221 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:28.229 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.257 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.257 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:38:28.257 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:28.292 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:38:28.354 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-24T16:38:28.354 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T16:38:28.355 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:28.398 INFO:teuthology.orchestra.run.trial151.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-24T16:38:28.400 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 33/39 2026-02-24T16:38:28.404 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:38 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:28.469 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:28.488 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:28.488 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:28.489 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:28.511 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:38:28.555 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:38:28.571 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:38:28.581 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:38:28.587 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:38:28.601 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:38:28.617 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:38:28.692 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:38:28.701 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:38:28.724 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:38:28.754 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:38:28.804 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:38:28.810 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:38:28.814 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:38:28.815 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:38:28.825 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:38 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:28.835 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.835 INFO:teuthology.orchestra.run.trial052.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:38:28.835 INFO:teuthology.orchestra.run.trial052.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:38:28.835 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:28.844 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.871 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:28.872 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:38:28.872 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:28.890 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:28.908 INFO:teuthology.orchestra.run.trial052.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:38:28.909 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:28.910 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:28.910 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:28.922 INFO:teuthology.orchestra.run.trial153.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:38:28.931 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd 2026-02-24T16:38:29.247 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:39 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:29.312 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:29.331 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:29.331 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:29.331 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:29.354 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd-classic 2026-02-24T16:38:29.465 INFO:teuthology.orchestra.run.trial051.stdout:(37/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 2.8 MB/s | 5.3 MB 00:01 2026-02-24T16:38:29.492 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:29.495 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:29.526 INFO:teuthology.orchestra.run.trial052.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:38:29.549 INFO:teuthology.orchestra.run.trial051.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-24T16:38:29.634 INFO:teuthology.orchestra.run.trial051.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.9 MB/s | 249 kB 00:00 2026-02-24T16:38:29.669 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:39 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:29.695 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:38:29.718 INFO:teuthology.orchestra.run.trial051.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 569 kB/s | 48 kB 00:00 2026-02-24T16:38:29.734 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:29.753 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:29.754 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 34/39 2026-02-24T16:38:29.754 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:29.754 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:29.757 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:29.758 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:38:29.760 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:29.775 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-fuse 2026-02-24T16:38:29.776 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:29.777 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:29.777 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:38:29.777 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:29.786 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:29.793 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 35/39 2026-02-24T16:38:29.794 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T16:38:29.794 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T16:38:29.794 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:29.797 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:29.802 INFO:teuthology.orchestra.run.trial051.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.1 MB/s | 177 kB 00:00 2026-02-24T16:38:29.808 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:29.808 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:29.808 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:38:29.808 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:29.808 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:29.809 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:29.863 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:29.864 INFO:teuthology.orchestra.run.trial153.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:29.874 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:38:29.891 INFO:teuthology.orchestra.run.trial051.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2026-02-24T16:38:29.898 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:38:29.899 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /sys 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /proc 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /mnt 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /var/tmp 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /home 2026-02-24T16:38:29.969 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /root 2026-02-24T16:38:29.970 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /tmp 2026-02-24T16:38:29.970 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:29.975 INFO:teuthology.orchestra.run.trial051.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 412 kB/s | 35 kB 00:00 2026-02-24T16:38:30.091 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:40 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:30.176 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:30.177 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:30.177 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 987 k 2026-02-24T16:38:30.177 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 2.9 M 2026-02-24T16:38:30.177 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:30.288 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:38:30.297 INFO:teuthology.orchestra.run.trial191.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 8.0 MB/s | 987 kB 00:00 2026-02-24T16:38:30.298 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:30.298 INFO:teuthology.orchestra.run.trial191.stdout:Total 8.0 MB/s | 987 kB 00:00 2026-02-24T16:38:30.298 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:30.302 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:30.303 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:30.337 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:30.337 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:30.352 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:38:30.354 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:30.370 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:30.371 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:30.371 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:38:30.371 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:30.381 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:30.390 INFO:teuthology.orchestra.run.trial051.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 15 MB/s | 6.1 MB 00:00 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:30.406 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:30.430 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:30.448 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:30.462 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:30.464 INFO:teuthology.orchestra.run.trial052.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:30.475 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:38:30.476 INFO:teuthology.orchestra.run.trial051.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.0 MB/s | 442 kB 00:00 2026-02-24T16:38:30.483 INFO:teuthology.orchestra.run.trial051.stdout:(46/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 876 kB/s | 3.0 MB 00:03 2026-02-24T16:38:30.501 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:38:30.502 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:30.553 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:30.568 INFO:teuthology.orchestra.run.trial051.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 3.1 MB/s | 267 kB 00:00 2026-02-24T16:38:30.577 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:30.578 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:30.578 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:30.578 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:30.578 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:30.578 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:30.579 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:30.593 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:30.593 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:30.642 INFO:teuthology.orchestra.run.trial051.stdout:(48/96): python3-packaging-20.9-5.el9.noarch.rp 471 kB/s | 77 kB 00:00 2026-02-24T16:38:30.650 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:30.650 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:30.650 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:30.652 INFO:teuthology.orchestra.run.trial051.stdout:(49/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.8 MB/s | 157 kB 00:00 2026-02-24T16:38:30.704 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:30.736 INFO:teuthology.orchestra.run.trial051.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 642 kB/s | 54 kB 00:00 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:30.766 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:30.794 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-volume 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:30.951 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:30.952 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:30.961 INFO:teuthology.orchestra.run.trial051.stdout:(51/96): python3-pyasn1-modules-0.4.8-7.el9.noa 867 kB/s | 277 kB 00:00 2026-02-24T16:38:31.021 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:31.021 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:31.021 INFO:teuthology.orchestra.run.trial183.stdout:Upgraded: 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:31.022 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:31.023 INFO:teuthology.orchestra.run.trial183.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.024 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:31.025 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:31.045 INFO:teuthology.orchestra.run.trial051.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2026-02-24T16:38:31.065 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-test 2026-02-24T16:38:31.117 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:41 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:31.182 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:31.201 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:31.202 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:31.202 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:31.223 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados-devel 2026-02-24T16:38:31.286 INFO:teuthology.orchestra.run.trial051.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 785 kB/s | 189 kB 00:00 2026-02-24T16:38:31.378 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:18 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:38:31.447 INFO:teuthology.orchestra.run.trial051.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 559 kB/s | 90 kB 00:00 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:38:31.462 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Install 10 Packages 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 80 M 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 331 M 2026-02-24T16:38:31.463 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:38:31.495 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:31.502 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:31.540 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:41 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:31.567 INFO:teuthology.orchestra.run.trial051.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 186 kB/s | 22 kB 00:00 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:31.626 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:31.627 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:31.627 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:31.627 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 123 k 2026-02-24T16:38:31.627 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 437 k 2026-02-24T16:38:31.627 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:31.715 INFO:teuthology.orchestra.run.trial191.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 1.4 MB/s | 123 kB 00:00 2026-02-24T16:38:31.715 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:31.715 INFO:teuthology.orchestra.run.trial191.stdout:Total 1.4 MB/s | 123 kB 00:00 2026-02-24T16:38:31.716 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:31.719 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:31.719 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:31.750 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:31.750 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:31.805 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:31.823 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:31.908 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:31.961 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:31.962 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:31.971 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:31.974 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:31.975 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:31.999 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs2 2026-02-24T16:38:32.037 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:32.045 INFO:teuthology.orchestra.run.trial051.stdout:(56/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 15 MB/s | 19 MB 00:01 2026-02-24T16:38:32.067 INFO:teuthology.orchestra.run.trial183.stdout:(1/10): libcephsqlite-20.3.0-5507.g618c318a.el9 1.8 MB/s | 159 kB 00:00 2026-02-24T16:38:32.080 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:38:32.081 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:32.088 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:32.095 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:32.102 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:32.114 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:32.272 INFO:teuthology.orchestra.run.trial051.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.4 MB/s | 551 kB 00:00 2026-02-24T16:38:32.304 INFO:teuthology.orchestra.run.trial051.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 605 kB/s | 19 kB 00:00 2026-02-24T16:38:32.314 INFO:teuthology.orchestra.run.trial183.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 92 kB/s | 23 kB 00:00 2026-02-24T16:38:32.317 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:42 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:32.337 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:32.338 INFO:teuthology.orchestra.run.trial051.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2026-02-24T16:38:32.339 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:32.360 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:32.360 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:32.360 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:38:32.361 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:32.361 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:32.361 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:32.374 INFO:teuthology.orchestra.run.trial051.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 15 MB/s | 548 kB 00:00 2026-02-24T16:38:32.382 INFO:teuthology.orchestra.run.trial191.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:32.401 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:32.402 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:32.402 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:32.407 INFO:teuthology.orchestra.run.trial051.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 900 kB/s | 29 kB 00:00 2026-02-24T16:38:32.425 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs-devel 2026-02-24T16:38:32.440 INFO:teuthology.orchestra.run.trial051.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.8 MB/s | 60 kB 00:00 2026-02-24T16:38:32.459 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:32.473 INFO:teuthology.orchestra.run.trial051.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2026-02-24T16:38:32.480 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:32.480 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:32.480 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:38:32.480 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:32.480 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:32.481 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:32.481 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:32.488 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:32.506 INFO:teuthology.orchestra.run.trial051.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 987 kB/s | 32 kB 00:00 2026-02-24T16:38:32.513 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:32.540 INFO:teuthology.orchestra.run.trial051.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 436 kB/s | 14 kB 00:00 2026-02-24T16:38:32.552 INFO:teuthology.orchestra.run.trial183.stdout:(3/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 396 kB/s | 94 kB 00:00 2026-02-24T16:38:32.555 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:32.556 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:32.569 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:32.570 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:32.575 INFO:teuthology.orchestra.run.trial051.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.9 MB/s | 173 kB 00:00 2026-02-24T16:38:32.614 INFO:teuthology.orchestra.run.trial051.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 8.8 MB/s | 358 kB 00:00 2026-02-24T16:38:32.622 INFO:teuthology.orchestra.run.trial051.stdout:(68/96): protobuf-compiler-3.14.0-17.el9.x86_64 817 kB/s | 862 kB 00:01 2026-02-24T16:38:32.632 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:32.649 INFO:teuthology.orchestra.run.trial051.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 7.1 MB/s | 254 kB 00:00 2026-02-24T16:38:32.674 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:38:32.675 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:32.684 INFO:teuthology.orchestra.run.trial051.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.1 MB/s | 144 kB 00:00 2026-02-24T16:38:32.698 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:32.698 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:32.698 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:38:32.698 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:32.699 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:32.699 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:32.710 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:32.719 INFO:teuthology.orchestra.run.trial051.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.0 MB/s | 139 kB 00:00 2026-02-24T16:38:32.722 INFO:teuthology.orchestra.run.trial183.stdout:(4/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.8 MB/s | 2.1 MB 00:00 2026-02-24T16:38:32.746 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:42 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:32.752 INFO:teuthology.orchestra.run.trial051.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.6 MB/s | 56 kB 00:00 2026-02-24T16:38:32.785 INFO:teuthology.orchestra.run.trial051.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 328 kB/s | 11 kB 00:00 2026-02-24T16:38:32.817 INFO:teuthology.orchestra.run.trial051.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 544 kB/s | 18 kB 00:00 2026-02-24T16:38:32.831 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:32.832 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:32.833 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:32.833 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 37 k 2026-02-24T16:38:32.833 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 171 k 2026-02-24T16:38:32.833 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:32.850 INFO:teuthology.orchestra.run.trial051.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 712 kB/s | 23 kB 00:00 2026-02-24T16:38:32.883 INFO:teuthology.orchestra.run.trial051.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 604 kB/s | 20 kB 00:00 2026-02-24T16:38:32.914 INFO:teuthology.orchestra.run.trial051.stdout:(77/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.0 MB/s | 2.0 MB 00:00 2026-02-24T16:38:32.915 INFO:teuthology.orchestra.run.trial051.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 597 kB/s | 19 kB 00:00 2026-02-24T16:38:32.934 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:32.935 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:32.947 INFO:teuthology.orchestra.run.trial051.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 790 kB/s | 26 kB 00:00 2026-02-24T16:38:32.958 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:32.958 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:32.958 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:38:32.959 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:32.959 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:32.959 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:32.982 INFO:teuthology.orchestra.run.trial051.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.2 MB/s | 79 kB 00:00 2026-02-24T16:38:32.984 INFO:teuthology.orchestra.run.trial051.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 15 MB/s | 1.0 MB 00:00 2026-02-24T16:38:33.016 INFO:teuthology.orchestra.run.trial051.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.5 MB/s | 86 kB 00:00 2026-02-24T16:38:33.017 INFO:teuthology.orchestra.run.trial051.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2026-02-24T16:38:33.049 INFO:teuthology.orchestra.run.trial051.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 490 kB/s | 16 kB 00:00 2026-02-24T16:38:33.051 INFO:teuthology.orchestra.run.trial051.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2026-02-24T16:38:33.054 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:33.075 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:33.076 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:33.076 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:38:33.076 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:33.076 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:33.076 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:33.077 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:33.083 INFO:teuthology.orchestra.run.trial051.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 917 kB/s | 31 kB 00:00 2026-02-24T16:38:33.083 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:33.086 INFO:teuthology.orchestra.run.trial051.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.3 MB/s | 188 kB 00:00 2026-02-24T16:38:33.095 INFO:teuthology.orchestra.run.trial191.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 141 kB/s | 37 kB 00:00 2026-02-24T16:38:33.096 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:33.096 INFO:teuthology.orchestra.run.trial191.stdout:Total 141 kB/s | 37 kB 00:00 2026-02-24T16:38:33.096 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:33.099 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:33.099 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:33.101 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:33.102 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:33.110 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:33.117 INFO:teuthology.orchestra.run.trial051.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2026-02-24T16:38:33.120 INFO:teuthology.orchestra.run.trial051.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.5 MB/s | 125 kB 00:00 2026-02-24T16:38:33.124 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:33.142 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:33.150 INFO:teuthology.orchestra.run.trial051.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2026-02-24T16:38:33.153 INFO:teuthology.orchestra.run.trial051.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.6 MB/s | 86 kB 00:00 2026-02-24T16:38:33.185 INFO:teuthology.orchestra.run.trial051.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2026-02-24T16:38:33.186 INFO:teuthology.orchestra.run.trial051.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.4 MB/s | 48 kB 00:00 2026-02-24T16:38:33.204 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:33.218 INFO:teuthology.orchestra.run.trial051.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 618 kB/s | 20 kB 00:00 2026-02-24T16:38:33.219 INFO:teuthology.orchestra.run.trial051.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 653 kB/s | 22 kB 00:00 2026-02-24T16:38:33.267 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:33.267 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:33.267 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:33.268 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.268 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:33.268 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:33.285 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:33.285 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:38:33.285 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:38:33.285 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:38:33.286 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:38:33.287 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:38:33.287 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:38:33.287 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:38:33.287 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:38:33.288 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:38:33.289 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:38:33.290 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:38:33.291 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:38:33.292 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:38:33.293 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:38:33.294 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:38:33.294 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:38:33.294 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:38:33.294 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:38:33.297 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados2 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.354 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.355 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:38:33.356 INFO:teuthology.orchestra.run.trial153.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:38:33.357 INFO:teuthology.orchestra.run.trial153.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:38:33.358 INFO:teuthology.orchestra.run.trial153.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:38:33.359 INFO:teuthology.orchestra.run.trial153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:38:33.360 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:38:33.361 INFO:teuthology.orchestra.run.trial153.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:33.362 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:33.411 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-base 2026-02-24T16:38:33.481 INFO:teuthology.orchestra.run.trial183.stdout:(5/10): ceph-test-20.3.0-5507.g618c318a.el9.x86 51 MB/s | 76 MB 00:01 2026-02-24T16:38:33.482 INFO:teuthology.orchestra.run.trial183.stdout:(6/10): libxslt-1.1.34-12.el9.x86_64.rpm 250 kB/s | 233 kB 00:00 2026-02-24T16:38:33.614 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:43 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:33.680 INFO:teuthology.orchestra.run.trial191.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:33.698 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:33.699 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:33.699 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:33.721 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librbd1 2026-02-24T16:38:33.724 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:34 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:33.752 INFO:teuthology.orchestra.run.trial183.stdout:(7/10): protobuf-3.14.0-17.el9.x86_64.rpm 998 kB/s | 1.0 MB 00:01 2026-02-24T16:38:33.789 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:33.808 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:33.808 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:33.808 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:33.828 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install cephadm 2026-02-24T16:38:33.877 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:33.877 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:38:33.877 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:38:33.877 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:38:33.878 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:38:33.881 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:38:33.882 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:38:33.883 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:38:33.884 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:38:33.885 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:38:33.910 INFO:teuthology.orchestra.run.trial183.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 774 kB/s | 122 kB 00:00 2026-02-24T16:38:33.953 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:38:33.953 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:33.953 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.954 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:38:33.955 INFO:teuthology.orchestra.run.trial052.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:38:33.956 INFO:teuthology.orchestra.run.trial052.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:38:33.957 INFO:teuthology.orchestra.run.trial052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:38:33.958 INFO:teuthology.orchestra.run.trial052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:38:33.959 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:38:33.960 INFO:teuthology.orchestra.run.trial052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:38:33.961 INFO:teuthology.orchestra.run.trial052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:38:33.961 INFO:teuthology.orchestra.run.trial052.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:38:33.961 INFO:teuthology.orchestra.run.trial052.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:38:33.961 INFO:teuthology.orchestra.run.trial052.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:33.961 INFO:teuthology.orchestra.run.trial052.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:38:33.962 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:38:33.963 INFO:teuthology.orchestra.run.trial052.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:38:33.964 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:33.965 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:33.965 INFO:teuthology.orchestra.run.trial183.stdout:(9/10): socat-1.7.4.1-8.el9.x86_64.rpm 639 kB/s | 303 kB 00:00 2026-02-24T16:38:34.017 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-base 2026-02-24T16:38:34.035 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:34.101 INFO:teuthology.orchestra.run.trial191.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:34.119 INFO:teuthology.orchestra.run.trial183.stdout:(10/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 100 kB/s | 64 kB 00:00 2026-02-24T16:38:34.120 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:34.120 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:34.120 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:34.124 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:34.124 INFO:teuthology.orchestra.run.trial183.stdout:Total 30 MB/s | 80 MB 00:02 2026-02-24T16:38:34.142 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rados 2026-02-24T16:38:34.144 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:35 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:34.192 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:38:34.202 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:38:34.202 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:38:34.208 INFO:teuthology.orchestra.run.trial153.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:34.227 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:34.227 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:34.227 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:34.248 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /sys 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /proc 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /mnt 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /var/tmp 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /home 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /root 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout:skipping the directory /tmp 2026-02-24T16:38:34.294 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:34.347 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:34.413 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:34.431 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:34.432 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:34.432 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:34.454 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install cephadm 2026-02-24T16:38:34.457 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:34.479 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:38:34.479 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:38:34.522 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:34.527 INFO:teuthology.orchestra.run.trial051.stdout:(96/96): python3-babel-2.9.1-2.el9.noarch.rpm 894 kB/s | 6.0 MB 00:06 2026-02-24T16:38:34.530 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:34.530 INFO:teuthology.orchestra.run.trial051.stdout:Total 5.9 MB/s | 123 MB 00:20 2026-02-24T16:38:34.540 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:34.541 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:34.541 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:34.561 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:35 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:34.562 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rgw 2026-02-24T16:38:34.645 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:34.645 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:34.646 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 147 k 2026-02-24T16:38:34.647 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 419 k 2026-02-24T16:38:34.647 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:34.732 INFO:teuthology.orchestra.run.trial153.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 1.7 MB/s | 147 kB 00:00 2026-02-24T16:38:34.733 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:34.733 INFO:teuthology.orchestra.run.trial153.stdout:Total 1.7 MB/s | 147 kB 00:00 2026-02-24T16:38:34.733 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:34.737 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:34.737 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:34.769 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:34.770 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:34.772 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:34.781 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:38:34.790 INFO:teuthology.orchestra.run.trial183.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:34.799 INFO:teuthology.orchestra.run.trial183.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:34.805 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:34.809 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:34.814 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:34.816 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:34.825 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:34.826 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:34.827 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:34.834 INFO:teuthology.orchestra.run.trial183.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:34.834 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:34.838 INFO:teuthology.orchestra.run.trial052.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:34.849 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:34.849 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:34.849 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:38:34.850 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:34.858 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:34.858 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:34.858 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:34.877 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:34.880 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:38:34.881 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 37/39 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T16:38:34.900 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:34.902 INFO:teuthology.orchestra.run.trial151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:34.914 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-24T16:38:34.914 INFO:teuthology.orchestra.run.trial151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:34.930 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:34.947 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:34.965 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:34.966 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:34.966 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:34.970 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:34.970 INFO:teuthology.orchestra.run.trial151.stdout:Creating group 'qat' with GID 992. 2026-02-24T16:38:34.970 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:34.970 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:34.970 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:34.987 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-cephfs 2026-02-24T16:38:35.154 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:35.155 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:35.155 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:35.155 INFO:teuthology.orchestra.run.trial153.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.155 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:35.155 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:35.184 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr 2026-02-24T16:38:35.192 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:35.267 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 1/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 3/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 4/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_6 5/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_6 6/39 2026-02-24T16:38:35.268 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 7/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_6 8/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 9/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9. 10/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x8 11/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 12/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 13/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 14/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 15/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-24T16:38:35.269 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-24T16:38:35.270 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librados2-2:20.3.0-5507.g618c318a.el9.x86_64 36/39 2026-02-24T16:38:35.271 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-24T16:38:35.272 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 38/39 2026-02-24T16:38:35.279 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:35.279 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:35.279 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repo Size 2026-02-24T16:38:35.279 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:35.279 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 147 k 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 419 k 2026-02-24T16:38:35.280 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:35.304 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:35.335 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-24T16:38:35.335 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:35.335 INFO:teuthology.orchestra.run.trial151.stdout:Upgraded: 2026-02-24T16:38:35.335 INFO:teuthology.orchestra.run.trial151.stdout: librados2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout: librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout: ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout: ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.336 INFO:teuthology.orchestra.run.trial151.stdout: ceph-radosgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: ceph-selinux-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-daemon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-proxy2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T16:38:35.337 INFO:teuthology.orchestra.run.trial151.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: libradosstriper1-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: librgw2-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T16:38:35.338 INFO:teuthology.orchestra.run.trial151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-ceph-argparse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-ceph-common-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.339 INFO:teuthology.orchestra.run.trial151.stdout: python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:35.340 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:38:35.367 INFO:teuthology.orchestra.run.trial052.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 1.7 MB/s | 147 kB 00:00 2026-02-24T16:38:35.367 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:35.367 INFO:teuthology.orchestra.run.trial052.stdout:Total 1.6 MB/s | 147 kB 00:00 2026-02-24T16:38:35.367 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:35.369 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:35.371 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:35.371 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:35.377 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-test 2026-02-24T16:38:35.388 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:35.389 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:35.389 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:35.402 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:35.402 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:35.411 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rbd 2026-02-24T16:38:35.458 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:35.460 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:35.470 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:35.470 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:35.482 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:35.482 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:35.482 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:38:35.483 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:35.497 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:36 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:35.563 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:35.582 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:35.583 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:35.583 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:35.604 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:38:35.703 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:00:23 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:38:35.729 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:35.786 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repository Size 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout: ceph-test x86_64 2:20.3.0-5507.g618c318a.el9 ceph 76 M 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout:Installing dependencies: 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T16:38:35.788 INFO:teuthology.orchestra.run.trial151.stdout: libcephsqlite x86_64 2:20.3.0-5507.g618c318a.el9 ceph 159 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout:Install 10 Packages 2026-02-24T16:38:35.789 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:35.790 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 80 M 2026-02-24T16:38:35.790 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 331 M 2026-02-24T16:38:35.790 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:38:35.794 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:35.803 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:35.813 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:35.813 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:35.814 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:35.829 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-mgr 2026-02-24T16:38:35.835 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-fuse 2026-02-24T16:38:35.916 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:36 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:35.982 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:36.001 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:36.001 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:36.002 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:36.005 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:36.007 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:38:36.010 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:38:36.022 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:38:36.032 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:38:36.065 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:38:36.066 INFO:teuthology.orchestra.run.trial151.stdout:(1/10): libcephsqlite-20.3.0-5507.g618c318a.el9 1.8 MB/s | 159 kB 00:00 2026-02-24T16:38:36.071 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:38:36.074 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:38:36.078 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:38:36.102 INFO:teuthology.orchestra.run.trial051.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:38:36.114 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:38:36.118 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:38:36.124 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:38:36.126 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:38:36.137 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:38:36.141 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:46 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:36.143 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:38:36.152 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:46 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:36.168 INFO:teuthology.orchestra.run.trial051.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:38:36.171 INFO:teuthology.orchestra.run.trial051.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:38:36.206 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:36.223 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:38:36.225 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:36.225 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:36.225 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:36.236 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:38:36.236 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:36.237 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:36.238 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:36.239 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:36.239 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 84 k 2026-02-24T16:38:36.239 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 209 k 2026-02-24T16:38:36.239 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:36.239 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:38:36.243 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:38:36.247 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:38:36.248 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:38:36.252 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:38:36.256 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:38:36.270 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:38:36.275 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:38:36.280 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:38:36.291 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:38:36.300 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:38:36.307 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:38:36.326 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:38:36.337 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:37 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:36.337 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:38:36.353 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:38:36.358 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:38:36.365 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:38:36.402 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:36.406 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:38:36.421 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:36.422 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:36.422 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:36.443 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:38:36.545 INFO:teuthology.orchestra.run.trial191.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 274 kB/s | 84 kB 00:00 2026-02-24T16:38:36.545 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:36.545 INFO:teuthology.orchestra.run.trial191.stdout:Total 273 kB/s | 84 kB 00:00 2026-02-24T16:38:36.546 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:36.549 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:36.550 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:36.565 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:46 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:36.581 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:36.581 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:36.622 INFO:teuthology.orchestra.run.trial151.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 41 kB/s | 23 kB 00:00 2026-02-24T16:38:36.631 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:36.634 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:36.650 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:36.650 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:36.651 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:36.651 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:36.671 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:38:36.734 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:38:36.744 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:38:36.747 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:36.748 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:38:36.754 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:38:36.756 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:38:36.761 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:37 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:36.784 INFO:teuthology.orchestra.run.trial051.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:38:36.810 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:36.810 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:36.810 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:36.810 INFO:teuthology.orchestra.run.trial191.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:36.810 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:36.811 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:36.825 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:36.829 INFO:teuthology.orchestra.run.trial051.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:38:36.838 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-mirror 2026-02-24T16:38:36.839 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:38:36.844 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:36.844 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:36.845 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:36.845 INFO:teuthology.orchestra.run.trial051.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:38:36.849 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:38:36.854 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:38:36.858 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:38:36.865 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:38:36.866 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:38:36.869 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:38:36.896 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:38:36.906 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:38:36.912 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:38:36.919 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:38:36.937 INFO:teuthology.orchestra.run.trial151.stdout:(3/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 298 kB/s | 94 kB 00:00 2026-02-24T16:38:36.954 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:38:36.986 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:46 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:37.051 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:37.070 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:37.070 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:37.070 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:37.092 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:38:37.158 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:37.181 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:38:37.181 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:38 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:37.205 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:38:37.209 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:38:37.241 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:38:37.242 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 2.9 M 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 11 M 2026-02-24T16:38:37.243 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:37.248 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:37.266 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:37.267 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:37.267 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:37.267 INFO:teuthology.orchestra.run.trial051.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:38:37.269 INFO:teuthology.orchestra.run.trial051.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:38:37.288 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd 2026-02-24T16:38:37.292 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:38:37.411 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:37.476 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:37.494 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:37.495 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:37.495 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:37.496 INFO:teuthology.orchestra.run.trial191.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 11 MB/s | 2.9 MB 00:00 2026-02-24T16:38:37.496 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:37.496 INFO:teuthology.orchestra.run.trial191.stdout:Total 11 MB/s | 2.9 MB 00:00 2026-02-24T16:38:37.496 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:37.499 INFO:teuthology.orchestra.run.trial151.stdout:(4/10): ceph-test-20.3.0-5507.g618c318a.el9.x86 50 MB/s | 76 MB 00:01 2026-02-24T16:38:37.500 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:37.500 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:37.516 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:38:37.541 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:37.541 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:37.604 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:38 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:37.618 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:38:37.670 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:37.688 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:37.689 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:37.689 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:37.692 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:38:37.710 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd-classic 2026-02-24T16:38:37.721 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:37.722 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:37.727 INFO:teuthology.orchestra.run.trial151.stdout:(5/10): libxslt-1.1.34-12.el9.x86_64.rpm 295 kB/s | 233 kB 00:00 2026-02-24T16:38:37.746 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:37.746 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:37.746 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:38:37.746 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:37.746 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:37.747 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:37.833 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:37.898 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:37.917 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:37.918 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:37.918 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:37.939 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-osd 2026-02-24T16:38:38.030 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:39 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:38.079 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:38.080 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:38.080 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:38.080 INFO:teuthology.orchestra.run.trial191.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:38.080 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:38.080 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:38.098 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:38.106 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-nbd 2026-02-24T16:38:38.117 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:38.117 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:38.117 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:38.138 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-fuse 2026-02-24T16:38:38.200 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:38.219 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:38.253 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:38.318 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:38.337 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:38.337 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:38.337 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:38.358 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-osd-classic 2026-02-24T16:38:38.367 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:38:38.384 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:38:38.394 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:38:38.400 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:38:38.414 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:38:38.420 INFO:teuthology.orchestra.run.trial151.stdout:(6/10): protobuf-3.14.0-17.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2026-02-24T16:38:38.426 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:38.430 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:38:38.454 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:39 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:38.506 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:38:38.509 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-24T16:38:38.510 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-24T16:38:38.511 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:38.511 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 172 k 2026-02-24T16:38:38.511 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 462 k 2026-02-24T16:38:38.511 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-24T16:38:38.515 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:38:38.522 INFO:teuthology.orchestra.run.trial151.stdout:(7/10): socat-1.7.4.1-8.el9.x86_64.rpm 383 kB/s | 303 kB 00:00 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:38.532 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:38.538 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:38.539 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:38.540 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 987 k 2026-02-24T16:38:38.540 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 2.9 M 2026-02-24T16:38:38.540 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:38.540 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:38:38.552 INFO:teuthology.orchestra.run.trial151.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.1 MB/s | 122 kB 00:00 2026-02-24T16:38:38.569 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:38.592 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:38.593 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:38.620 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:38:38.626 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:38:38.629 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:38:38.631 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph 2026-02-24T16:38:38.632 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:38:38.649 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:38.649 INFO:teuthology.orchestra.run.trial051.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:38:38.649 INFO:teuthology.orchestra.run.trial051.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:38:38.649 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:38.656 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:38.675 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:38.681 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:38.681 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:38:38.682 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:38.716 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:38:38.741 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:38.759 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:38.760 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:38.760 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:38.781 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-fuse 2026-02-24T16:38:38.875 INFO:teuthology.orchestra.run.trial191.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 473 kB/s | 172 kB 00:00 2026-02-24T16:38:38.875 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:38.875 INFO:teuthology.orchestra.run.trial191.stdout:Total 472 kB/s | 172 kB 00:00 2026-02-24T16:38:38.876 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-24T16:38:38.880 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-24T16:38:38.880 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-24T16:38:38.913 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-24T16:38:38.913 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-24T16:38:38.949 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:25 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:38:38.969 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-24T16:38:38.987 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.022 INFO:teuthology.orchestra.run.trial151.stdout:(9/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 707 kB/s | 2.1 MB 00:03 2026-02-24T16:38:39.035 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:39.038 INFO:teuthology.orchestra.run.trial183.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:39.039 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:39.040 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:39.041 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:39.042 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:39.043 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout:Install 96 Packages 2026-02-24T16:38:39.044 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:39.045 INFO:teuthology.orchestra.run.trial151.stdout:(10/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 103 kB/s | 64 kB 00:00 2026-02-24T16:38:39.046 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:39.046 INFO:teuthology.orchestra.run.trial151.stdout:Total 25 MB/s | 80 MB 00:03 2026-02-24T16:38:39.046 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 123 M 2026-02-24T16:38:39.046 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 656 M 2026-02-24T16:38:39.046 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:38:39.055 INFO:teuthology.orchestra.run.trial153.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.9 MB/s | 987 kB 00:00 2026-02-24T16:38:39.055 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:39.055 INFO:teuthology.orchestra.run.trial153.stdout:Total 1.9 MB/s | 987 kB 00:00 2026-02-24T16:38:39.055 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:39.060 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:39.060 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:39.078 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.095 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:39.095 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:39.099 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:49 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:39.112 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:38:39.124 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:38:39.124 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:38:39.141 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:39.141 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-24T16:38:39.141 INFO:teuthology.orchestra.run.trial191.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:39.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:38:39.142 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:39.172 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install nvme-cli 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repo Size 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:39.183 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:39.184 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 987 k 2026-02-24T16:38:39.184 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 2.9 M 2026-02-24T16:38:39.184 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:39.186 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:39.204 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.334 INFO:teuthology.orchestra.run.trial051.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:38:39.402 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:38:39.402 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:38:39.470 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.491 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:49 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:39.532 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:39.553 INFO:teuthology.orchestra.run.trial183.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 31 kB/s | 6.5 kB 00:00 2026-02-24T16:38:39.556 INFO:teuthology.orchestra.run.trial191.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:38:39.560 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-volume 2026-02-24T16:38:39.574 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-24T16:38:39.575 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-24T16:38:39.575 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-24T16:38:39.599 DEBUG:teuthology.parallel:result is None 2026-02-24T16:38:39.703 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:38:39.712 INFO:teuthology.orchestra.run.trial151.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-24T16:38:39.721 INFO:teuthology.orchestra.run.trial151.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-24T16:38:39.723 INFO:teuthology.orchestra.run.trial052.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.8 MB/s | 987 kB 00:00 2026-02-24T16:38:39.724 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:39.724 INFO:teuthology.orchestra.run.trial052.stdout:Total 1.8 MB/s | 987 kB 00:00 2026-02-24T16:38:39.724 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:39.726 INFO:teuthology.orchestra.run.trial151.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-24T16:38:39.729 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:39.729 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:39.730 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-24T16:38:39.735 INFO:teuthology.orchestra.run.trial151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-24T16:38:39.737 INFO:teuthology.orchestra.run.trial151.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-24T16:38:39.746 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-24T16:38:39.755 INFO:teuthology.orchestra.run.trial151.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-24T16:38:39.755 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:39.764 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:39.765 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:39.857 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:39.876 INFO:teuthology.orchestra.run.trial052.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:39.880 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:40 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:39.946 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:39.964 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:39.965 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:39.965 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:39.987 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados-devel 2026-02-24T16:38:40.097 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:38:40.099 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:40.162 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:38:40.162 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:40.163 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:40.163 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:40.163 INFO:teuthology.orchestra.run.trial052.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:40.163 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:40.163 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:40.164 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:40.178 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:40.178 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:40.178 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:38:40.178 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:40.188 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:40.191 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install ceph-volume 2026-02-24T16:38:40.197 INFO:teuthology.orchestra.run.trial183.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 1.1 MB/s | 950 kB 00:00 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:40.212 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:40.267 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:40.269 INFO:teuthology.orchestra.run.trial051.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:40.278 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:38:40.302 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:38:40.303 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:40.306 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:41 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:40.307 INFO:teuthology.orchestra.run.trial183.stdout:(3/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 1.5 MB/s | 167 kB 00:00 2026-02-24T16:38:40.393 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:40.393 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:40.394 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-24T16:38:40.394 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:40.394 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:40.394 INFO:teuthology.orchestra.run.trial153.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:38:40.394 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 123 k 2026-02-24T16:38:40.395 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 437 k 2026-02-24T16:38:40.396 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:40.511 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:50 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:40.560 INFO:teuthology.orchestra.run.trial153.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 732 kB/s | 123 kB 00:00 2026-02-24T16:38:40.561 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:40.562 INFO:teuthology.orchestra.run.trial153.stdout:Total 723 kB/s | 123 kB 00:00 2026-02-24T16:38:40.562 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:40.565 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:40.565 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:40.576 INFO:teuthology.orchestra.run.trial183.stdout:(4/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 1.9 MB/s | 2.4 MB 00:01 2026-02-24T16:38:40.576 INFO:teuthology.orchestra.run.trial052.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:40.595 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:40.596 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:40.597 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:40.597 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:40.598 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:40.618 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install librados-devel 2026-02-24T16:38:40.652 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:40.669 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:40.756 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:40.818 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:40.818 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:40.818 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:40.819 INFO:teuthology.orchestra.run.trial153.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:40.819 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:40.819 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:40.847 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs2 2026-02-24T16:38:40.934 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:50 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:41.019 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:41.019 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repo Size 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:41.020 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:41.021 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 123 k 2026-02-24T16:38:41.021 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 437 k 2026-02-24T16:38:41.021 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:41.134 INFO:teuthology.orchestra.run.trial183.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 3.2 MB/s | 5.0 MB 00:01 2026-02-24T16:38:41.146 INFO:teuthology.orchestra.run.trial183.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 4.8 MB/s | 57 kB 00:00 2026-02-24T16:38:41.150 INFO:teuthology.orchestra.run.trial052.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 944 kB/s | 123 kB 00:00 2026-02-24T16:38:41.150 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:41.150 INFO:teuthology.orchestra.run.trial052.stdout:Total 937 kB/s | 123 kB 00:00 2026-02-24T16:38:41.150 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:41.153 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:41.153 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:41.159 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:42 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:41.169 INFO:teuthology.orchestra.run.trial183.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 7.8 MB/s | 184 kB 00:00 2026-02-24T16:38:41.185 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:41.186 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:41.224 INFO:teuthology.orchestra.run.trial153.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:41.240 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:41.244 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:41.244 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:41.244 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:41.258 INFO:teuthology.orchestra.run.trial052.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.265 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs-devel 2026-02-24T16:38:41.344 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.408 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.408 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:41.408 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:41.408 INFO:teuthology.orchestra.run.trial052.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:41.408 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:41.409 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:41.438 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install libcephfs2 2026-02-24T16:38:41.581 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:42 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:41.666 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 37 k 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 171 k 2026-02-24T16:38:41.667 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:41.730 INFO:teuthology.orchestra.run.trial153.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 594 kB/s | 37 kB 00:00 2026-02-24T16:38:41.731 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:41.731 INFO:teuthology.orchestra.run.trial153.stdout:Total 581 kB/s | 37 kB 00:00 2026-02-24T16:38:41.732 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:41.734 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:41.734 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:41.736 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:41.736 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:41.754 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:51 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:41.758 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:41.775 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.808 INFO:teuthology.orchestra.run.trial183.stdout:(8/96): ceph-osd-classic-20.3.0-5507.g618c318a. 10 MB/s | 16 MB 00:01 2026-02-24T16:38:41.819 INFO:teuthology.orchestra.run.trial052.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:41.837 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:41.838 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:41.838 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:41.839 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.859 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install libcephfs-devel 2026-02-24T16:38:41.883 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:41.890 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:41.902 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:41.931 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados2 2026-02-24T16:38:42.108 INFO:teuthology.orchestra.run.trial183.stdout:(9/96): ceph-osd-crimson-20.3.0-5507.g618c318a. 12 MB/s | 19 MB 00:01 2026-02-24T16:38:42.120 INFO:teuthology.orchestra.run.trial183.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 1.9 MB/s | 22 kB 00:00 2026-02-24T16:38:42.143 INFO:teuthology.orchestra.run.trial183.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5507.g618 13 MB/s | 298 kB 00:00 2026-02-24T16:38:42.155 INFO:teuthology.orchestra.run.trial183.stdout:(12/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 4.3 MB/s | 50 kB 00:00 2026-02-24T16:38:42.166 INFO:teuthology.orchestra.run.trial183.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5507.g61 1.5 MB/s | 17 kB 00:00 2026-02-24T16:38:42.176 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:42.189 INFO:teuthology.orchestra.run.trial183.stdout:(14/96): ceph-volume-20.3.0-5507.g618c318a.el9. 13 MB/s | 301 kB 00:00 2026-02-24T16:38:42.247 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:43 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:42.256 INFO:teuthology.orchestra.run.trial183.stdout:(15/96): cephadm-20.3.0-5507.g618c318a.el9.noar 15 MB/s | 1.0 MB 00:00 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repo Size 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:42.260 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 37 k 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 171 k 2026-02-24T16:38:42.261 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:42.273 INFO:teuthology.orchestra.run.trial183.stdout:(16/96): ceph-mgr-diskprediction-local-20.3.0-5 16 MB/s | 7.4 MB 00:00 2026-02-24T16:38:42.312 INFO:teuthology.orchestra.run.trial153.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:42.325 INFO:teuthology.orchestra.run.trial052.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 586 kB/s | 37 kB 00:00 2026-02-24T16:38:42.326 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:42.326 INFO:teuthology.orchestra.run.trial052.stdout:Total 574 kB/s | 37 kB 00:00 2026-02-24T16:38:42.326 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:42.328 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:42.328 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:42.330 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:42.330 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:42.332 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:42.332 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:42.332 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:42.344 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:42.345 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:42.353 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librbd1 2026-02-24T16:38:42.354 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:42.358 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:42.358 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:42.370 INFO:teuthology.orchestra.run.trial052.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:42.421 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:42.429 INFO:teuthology.orchestra.run.trial183.stdout:(17/96): ceph-mgr-dashboard-20.3.0-5507.g618c31 12 MB/s | 15 MB 00:01 2026-02-24T16:38:42.436 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:42.463 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:38:42.464 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:42.485 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:42.486 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:42.486 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:38:42.486 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:42.486 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:42.486 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:42.497 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:42.499 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:42.499 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:42.499 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:42.499 INFO:teuthology.orchestra.run.trial052.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:42.500 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:42.500 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:42.527 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install librados2 2026-02-24T16:38:42.674 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:43 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:42.718 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:42.719 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:42.739 INFO:teuthology.orchestra.run.trial153.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:42.740 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:42.740 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:42.740 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:38:42.740 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:42.740 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:42.741 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:42.757 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:42.758 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:42.758 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:42.779 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rados 2026-02-24T16:38:42.836 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:42.844 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:42.857 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:42.857 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:42.857 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:38:42.857 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:42.858 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:42.858 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:42.858 INFO:teuthology.orchestra.run.trial183.stdout:(18/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 69 kB/s | 40 kB 00:00 2026-02-24T16:38:42.859 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:42.865 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:42.892 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:42.909 INFO:teuthology.orchestra.run.trial052.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:42.928 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:42.928 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:42.928 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:42.950 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install librbd1 2026-02-24T16:38:43.060 INFO:teuthology.orchestra.run.trial183.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 114 kB/s | 72 kB 00:00 2026-02-24T16:38:43.081 INFO:teuthology.orchestra.run.trial183.stdout:(20/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 425 kB/s | 351 kB 00:00 2026-02-24T16:38:43.096 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:43.111 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 9/10 2026-02-24T16:38:43.130 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:43.161 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:43.180 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:43.180 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:43.181 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:43.202 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rgw 2026-02-24T16:38:43.269 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:43.334 INFO:teuthology.orchestra.run.trial052.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:43.353 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:43.353 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:43.354 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:43.376 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install python3-rados 2026-02-24T16:38:43.451 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 10/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 1/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-24T16:38:43.452 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-24T16:38:43.453 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: ceph-test-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: libcephsqlite-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.512 INFO:teuthology.orchestra.run.trial151.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:43.513 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:38:43.518 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:43.550 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph 2026-02-24T16:38:43.583 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:43.602 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:43.603 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:43.603 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:43.631 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-cephfs 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:38:43.632 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:38:43.633 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:38:43.634 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:38:43.635 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:38:43.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:38:43.637 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:38:43.638 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:38:43.639 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:38:43.640 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:38:43.698 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:38:43.698 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:43.698 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:43.698 INFO:teuthology.orchestra.run.trial051.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.699 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:38:43.700 INFO:teuthology.orchestra.run.trial051.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:38:43.701 INFO:teuthology.orchestra.run.trial051.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:38:43.702 INFO:teuthology.orchestra.run.trial051.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:43.703 INFO:teuthology.orchestra.run.trial051.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:38:43.704 INFO:teuthology.orchestra.run.trial051.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:38:43.705 INFO:teuthology.orchestra.run.trial051.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:43.706 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:43.707 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:43.754 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-base 2026-02-24T16:38:43.757 INFO:teuthology.orchestra.run.trial183.stdout:(21/96): libgfortran-11.5.0-14.el9.x86_64.rpm 883 kB/s | 794 kB 00:00 2026-02-24T16:38:43.766 INFO:teuthology.orchestra.run.trial052.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:43.785 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:43.786 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:43.786 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:43.791 INFO:teuthology.orchestra.run.trial183.stdout:(22/96): libquadmath-11.5.0-14.el9.x86_64.rpm 252 kB/s | 184 kB 00:00 2026-02-24T16:38:43.808 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install python3-rgw 2026-02-24T16:38:43.850 INFO:teuthology.orchestra.run.trial183.stdout:(23/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 329 kB/s | 253 kB 00:00 2026-02-24T16:38:43.862 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:00:31 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:38:43.947 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:38:43.950 INFO:teuthology.orchestra.run.trial151.stdout:======================================================================================= 2026-02-24T16:38:43.950 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repository Size 2026-02-24T16:38:43.950 INFO:teuthology.orchestra.run.trial151.stdout:======================================================================================= 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph x86_64 2:20.3.0-5507.g618c318a.el9 ceph 6.5 k 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout:Installing dependencies: 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 57 k 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mds x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.4 M 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr x86_64 2:20.3.0-5507.g618c318a.el9 ceph 950 k 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 298 k 2026-02-24T16:38:43.951 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mon x86_64 2:20.3.0-5507.g618c318a.el9 ceph 5.0 M 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 167 k 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd-classic x86_64 2:20.3.0-5507.g618c318a.el9 ceph 16 M 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd-crimson x86_64 2:20.3.0-5507.g618c318a.el9 ceph 19 M 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 17 k 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 1.0 M 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:38:43.952 INFO:teuthology.orchestra.run.trial151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:38:43.953 INFO:teuthology.orchestra.run.trial151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:38:43.954 INFO:teuthology.orchestra.run.trial151.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:38:43.955 INFO:teuthology.orchestra.run.trial151.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:38:43.956 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T16:38:43.957 INFO:teuthology.orchestra.run.trial151.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout:Installing weak dependencies: 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 184 k 2026-02-24T16:38:43.958 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 15 M 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 7.4 M 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 22 k 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-rook noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 50 k 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: ceph-volume noarch 2:20.3.0-5507.g618c318a.el9 ceph-noarch 301 k 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout:======================================================================================= 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout:Install 96 Packages 2026-02-24T16:38:43.959 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:38:43.960 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:43.961 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 123 M 2026-02-24T16:38:43.961 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 656 M 2026-02-24T16:38:43.961 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:38:44.022 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:44.041 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:44.042 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:44.042 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:44.063 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rbd 2026-02-24T16:38:44.068 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:44.128 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:54 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:44.133 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:44.152 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:44.153 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:44.153 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:44.175 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install cephadm 2026-02-24T16:38:44.193 INFO:teuthology.orchestra.run.trial052.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:44.212 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:44.213 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:44.213 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:44.234 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install python3-cephfs 2026-02-24T16:38:44.382 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:44.448 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:44.461 INFO:teuthology.orchestra.run.trial183.stdout:(24/96): python3-ply-3.11-14.el9.noarch.rpm 159 kB/s | 106 kB 00:00 2026-02-24T16:38:44.466 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:44.466 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:44.466 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:44.487 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:44.488 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-fuse 2026-02-24T16:38:44.552 INFO:teuthology.orchestra.run.trial051.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:44.553 INFO:teuthology.orchestra.run.trial183.stdout:(25/96): python3-pycparser-2.20-6.el9.noarch.rp 192 kB/s | 135 kB 00:00 2026-02-24T16:38:44.556 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:54 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:44.571 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:44.571 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:44.571 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:44.593 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:38:44.622 INFO:teuthology.orchestra.run.trial052.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:44.640 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:44.641 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:44.641 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:44.662 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install python3-rbd 2026-02-24T16:38:44.708 INFO:teuthology.orchestra.run.trial183.stdout:(26/96): python3-cryptography-36.0.1-5.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2026-02-24T16:38:44.807 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:44.838 INFO:teuthology.orchestra.run.trial151.stdout:(1/96): ceph-20.3.0-5507.g618c318a.el9.x86_64.r 58 kB/s | 6.5 kB 00:00 2026-02-24T16:38:44.892 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:44.892 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:44.892 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:44.892 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 84 k 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 209 k 2026-02-24T16:38:44.893 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:44.909 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:44.986 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:54 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:44.992 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:44.993 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:44.993 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-24T16:38:44.993 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 147 k 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 419 k 2026-02-24T16:38:44.994 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:45.053 INFO:teuthology.orchestra.run.trial052.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:45.072 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:45.073 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:45.073 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:45.095 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install rbd-fuse 2026-02-24T16:38:45.127 INFO:teuthology.orchestra.run.trial153.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 358 kB/s | 84 kB 00:00 2026-02-24T16:38:45.128 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:45.128 INFO:teuthology.orchestra.run.trial153.stdout:Total 357 kB/s | 84 kB 00:00 2026-02-24T16:38:45.128 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:45.132 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:45.132 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:45.164 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:45.164 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:45.172 INFO:teuthology.orchestra.run.trial183.stdout:(27/96): python3-pyparsing-2.4.7-9.el9.noarch.r 211 kB/s | 150 kB 00:00 2026-02-24T16:38:45.217 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:45.234 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.253 INFO:teuthology.orchestra.run.trial183.stdout:(28/96): python3-requests-2.25.1-10.el9.noarch. 181 kB/s | 126 kB 00:00 2026-02-24T16:38:45.277 INFO:teuthology.orchestra.run.trial051.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 517 kB/s | 147 kB 00:00 2026-02-24T16:38:45.278 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:45.278 INFO:teuthology.orchestra.run.trial051.stdout:Total 516 kB/s | 147 kB 00:00 2026-02-24T16:38:45.278 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:45.283 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:45.283 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:45.315 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:45.315 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:45.323 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.371 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:45.372 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:45.375 INFO:teuthology.orchestra.run.trial151.stdout:(2/96): ceph-mgr-20.3.0-5507.g618c318a.el9.x86_ 1.4 MB/s | 950 kB 00:00 2026-02-24T16:38:45.385 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.385 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:45.385 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:45.386 INFO:teuthology.orchestra.run.trial153.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:45.386 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:45.386 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:45.396 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:45.396 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:45.396 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:38:45.396 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:45.410 INFO:teuthology.orchestra.run.trial151.stdout:(3/96): ceph-osd-20.3.0-5507.g618c318a.el9.x86_ 4.7 MB/s | 167 kB 00:00 2026-02-24T16:38:45.414 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-mirror 2026-02-24T16:38:45.415 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:45.455 INFO:teuthology.orchestra.run.trial183.stdout:(29/96): python3-urllib3-1.26.5-7.el9.noarch.rp 292 kB/s | 218 kB 00:00 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:45.501 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:45.502 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 84 k 2026-02-24T16:38:45.502 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 209 k 2026-02-24T16:38:45.502 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:45.607 INFO:teuthology.orchestra.run.trial151.stdout:(4/96): ceph-mds-20.3.0-5507.g618c318a.el9.x86_ 2.7 MB/s | 2.4 MB 00:00 2026-02-24T16:38:45.664 INFO:teuthology.orchestra.run.trial052.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 519 kB/s | 84 kB 00:00 2026-02-24T16:38:45.665 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:45.665 INFO:teuthology.orchestra.run.trial052.stdout:Total 516 kB/s | 84 kB 00:00 2026-02-24T16:38:45.665 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:45.669 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:45.669 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:45.702 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:45.702 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:45.714 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:45.715 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:45.715 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:45.715 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:45.715 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:45.715 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:45.716 INFO:teuthology.orchestra.run.trial183.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 114 kB/s | 30 kB 00:00 2026-02-24T16:38:45.733 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:46 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:45.742 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr 2026-02-24T16:38:45.754 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:45.761 INFO:teuthology.orchestra.run.trial151.stdout:(5/96): ceph-mon-20.3.0-5507.g618c318a.el9.x86_ 5.4 MB/s | 5.0 MB 00:00 2026-02-24T16:38:45.772 INFO:teuthology.orchestra.run.trial052.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.793 INFO:teuthology.orchestra.run.trial151.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5507.g61 1.8 MB/s | 57 kB 00:00 2026-02-24T16:38:45.817 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:45.817 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:45.817 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-24T16:38:45.817 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:45.817 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 2.9 M 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 11 M 2026-02-24T16:38:45.818 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:45.829 INFO:teuthology.orchestra.run.trial151.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5507.g618c318a. 5.1 MB/s | 184 kB 00:00 2026-02-24T16:38:45.857 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.907 INFO:teuthology.orchestra.run.trial183.stdout:(31/96): unzip-6.0-59.el9.x86_64.rpm 248 kB/s | 182 kB 00:00 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:45.921 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:45.950 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install rbd-mirror 2026-02-24T16:38:46.027 INFO:teuthology.orchestra.run.trial183.stdout:(32/96): zip-3.0-35.el9.x86_64.rpm 343 kB/s | 266 kB 00:00 2026-02-24T16:38:46.060 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:46.128 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:46.138 INFO:teuthology.orchestra.run.trial183.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 64 kB/s | 15 kB 00:00 2026-02-24T16:38:46.147 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:46.148 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:46.148 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:46.168 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:38:46.273 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:56 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:46.307 INFO:teuthology.orchestra.run.trial183.stdout:(34/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.1 MB/s | 3.0 MB 00:00 2026-02-24T16:38:46.356 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout: Package Arch Version Repo Size 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:46.357 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:46.358 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:46.358 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 2.9 M 2026-02-24T16:38:46.358 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 11 M 2026-02-24T16:38:46.358 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:46.385 INFO:teuthology.orchestra.run.trial183.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 537 kB/s | 42 kB 00:00 2026-02-24T16:38:46.462 INFO:teuthology.orchestra.run.trial183.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 582 kB/s | 188 kB 00:00 2026-02-24T16:38:46.482 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:46.523 INFO:teuthology.orchestra.run.trial183.stdout:(37/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 497 kB/s | 246 kB 00:00 2026-02-24T16:38:46.547 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:46.565 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:46.566 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:46.566 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:46.587 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:38:46.592 INFO:teuthology.orchestra.run.trial183.stdout:(38/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 26 MB/s | 5.3 MB 00:00 2026-02-24T16:38:46.657 INFO:teuthology.orchestra.run.trial183.stdout:(39/96): python3-devel-3.9.25-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2026-02-24T16:38:46.673 INFO:teuthology.orchestra.run.trial183.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 3.0 MB/s | 249 kB 00:00 2026-02-24T16:38:46.743 INFO:teuthology.orchestra.run.trial183.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 553 kB/s | 48 kB 00:00 2026-02-24T16:38:46.753 INFO:teuthology.orchestra.run.trial183.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.2 MB/s | 177 kB 00:00 2026-02-24T16:38:46.831 INFO:teuthology.orchestra.run.trial183.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 444 kB/s | 35 kB 00:00 2026-02-24T16:38:46.907 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:46.953 INFO:teuthology.orchestra.run.trial183.stdout:(44/96): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2026-02-24T16:38:46.972 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:46.991 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:46.991 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:46.991 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:46.994 INFO:teuthology.orchestra.run.trial151.stdout:(8/96): ceph-mgr-dashboard-20.3.0-5507.g618c318 13 MB/s | 15 MB 00:01 2026-02-24T16:38:47.001 INFO:teuthology.orchestra.run.trial183.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2026-02-24T16:38:47.012 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:38:47.022 INFO:teuthology.orchestra.run.trial183.stdout:(46/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 32 MB/s | 6.1 MB 00:00 2026-02-24T16:38:47.035 INFO:teuthology.orchestra.run.trial183.stdout:(47/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.3 MB/s | 442 kB 00:00 2026-02-24T16:38:47.082 INFO:teuthology.orchestra.run.trial183.stdout:(48/96): python3-packaging-20.9-5.el9.noarch.rp 958 kB/s | 77 kB 00:00 2026-02-24T16:38:47.103 INFO:teuthology.orchestra.run.trial183.stdout:(49/96): python3-protobuf-3.14.0-17.el9.noarch. 3.2 MB/s | 267 kB 00:00 2026-02-24T16:38:47.115 INFO:teuthology.orchestra.run.trial183.stdout:(50/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.9 MB/s | 157 kB 00:00 2026-02-24T16:38:47.170 INFO:teuthology.orchestra.run.trial183.stdout:(51/96): python3-pyasn1-modules-0.4.8-7.el9.noa 3.1 MB/s | 277 kB 00:00 2026-02-24T16:38:47.182 INFO:teuthology.orchestra.run.trial183.stdout:(52/96): python3-requests-oauthlib-1.3.0-12.el9 679 kB/s | 54 kB 00:00 2026-02-24T16:38:47.237 INFO:teuthology.orchestra.run.trial151.stdout:(9/96): ceph-osd-classic-20.3.0-5507.g618c318a. 8.6 MB/s | 16 MB 00:01 2026-02-24T16:38:47.249 INFO:teuthology.orchestra.run.trial183.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 528 kB/s | 42 kB 00:00 2026-02-24T16:38:47.261 INFO:teuthology.orchestra.run.trial183.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.3 MB/s | 189 kB 00:00 2026-02-24T16:38:47.275 INFO:teuthology.orchestra.run.trial151.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5507.g618c31 580 kB/s | 22 kB 00:00 2026-02-24T16:38:47.308 INFO:teuthology.orchestra.run.trial183.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 475 kB/s | 22 kB 00:00 2026-02-24T16:38:47.322 INFO:teuthology.orchestra.run.trial153.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.9 MB/s | 2.9 MB 00:01 2026-02-24T16:38:47.323 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:47.323 INFO:teuthology.orchestra.run.trial153.stdout:Total 1.9 MB/s | 2.9 MB 00:01 2026-02-24T16:38:47.323 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:47.328 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:47.328 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:47.329 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:56 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:47.329 INFO:teuthology.orchestra.run.trial183.stdout:(56/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-24T16:38:47.338 INFO:teuthology.orchestra.run.trial151.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5507.g618 4.7 MB/s | 298 kB 00:00 2026-02-24T16:38:47.369 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:47.369 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:47.394 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:47.394 INFO:teuthology.orchestra.run.trial183.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 9.8 MB/s | 862 kB 00:00 2026-02-24T16:38:47.407 INFO:teuthology.orchestra.run.trial151.stdout:(12/96): ceph-osd-crimson-20.3.0-5507.g618c318a 11 MB/s | 19 MB 00:01 2026-02-24T16:38:47.407 INFO:teuthology.orchestra.run.trial151.stdout:(13/96): ceph-mgr-rook-20.3.0-5507.g618c318a.el 721 kB/s | 50 kB 00:00 2026-02-24T16:38:47.412 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:47.413 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:47.413 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:47.416 INFO:teuthology.orchestra.run.trial151.stdout:(14/96): ceph-mgr-diskprediction-local-20.3.0-5 18 MB/s | 7.4 MB 00:00 2026-02-24T16:38:47.419 INFO:teuthology.orchestra.run.trial151.stdout:(15/96): ceph-prometheus-alerts-20.3.0-5507.g61 1.5 MB/s | 17 kB 00:00 2026-02-24T16:38:47.420 INFO:teuthology.orchestra.run.trial151.stdout:(16/96): ceph-volume-20.3.0-5507.g618c318a.el9. 24 MB/s | 301 kB 00:00 2026-02-24T16:38:47.434 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:38:47.437 INFO:teuthology.orchestra.run.trial151.stdout:(17/96): cephadm-20.3.0-5507.g618c318a.el9.noar 51 MB/s | 1.0 MB 00:00 2026-02-24T16:38:47.472 INFO:teuthology.orchestra.run.trial183.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 249 kB/s | 19 kB 00:00 2026-02-24T16:38:47.493 INFO:teuthology.orchestra.run.trial052.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 2.5 MB/s | 2.9 MB 00:01 2026-02-24T16:38:47.494 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:47.494 INFO:teuthology.orchestra.run.trial052.stdout:Total 2.5 MB/s | 2.9 MB 00:01 2026-02-24T16:38:47.494 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:47.499 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:47.499 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:47.523 INFO:teuthology.orchestra.run.trial183.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.8 MB/s | 551 kB 00:00 2026-02-24T16:38:47.539 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:47.540 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:47.547 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:47.549 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:47.553 INFO:teuthology.orchestra.run.trial183.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2026-02-24T16:38:47.555 INFO:teuthology.orchestra.run.trial183.stdout:(61/96): python3-asyncssh-2.13.2-5.el9.noarch.r 17 MB/s | 548 kB 00:00 2026-02-24T16:38:47.572 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:47.573 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:47.573 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:38:47.573 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:47.573 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:47.573 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:47.582 INFO:teuthology.orchestra.run.trial183.stdout:(62/96): python3-autocommand-2.2.2-8.el9.noarch 1.0 MB/s | 29 kB 00:00 2026-02-24T16:38:47.584 INFO:teuthology.orchestra.run.trial183.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 2.0 MB/s | 60 kB 00:00 2026-02-24T16:38:47.611 INFO:teuthology.orchestra.run.trial183.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.5 MB/s | 43 kB 00:00 2026-02-24T16:38:47.614 INFO:teuthology.orchestra.run.trial183.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 1.1 MB/s | 32 kB 00:00 2026-02-24T16:38:47.640 INFO:teuthology.orchestra.run.trial183.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 483 kB/s | 14 kB 00:00 2026-02-24T16:38:47.646 INFO:teuthology.orchestra.run.trial183.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.4 MB/s | 173 kB 00:00 2026-02-24T16:38:47.679 INFO:teuthology.orchestra.run.trial183.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 7.4 MB/s | 254 kB 00:00 2026-02-24T16:38:47.707 INFO:teuthology.orchestra.run.trial183.stdout:(69/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 33 MB/s | 19 MB 00:00 2026-02-24T16:38:47.708 INFO:teuthology.orchestra.run.trial183.stdout:(70/96): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2026-02-24T16:38:47.717 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:47.719 INFO:teuthology.orchestra.run.trial052.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:47.740 INFO:teuthology.orchestra.run.trial183.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.2 MB/s | 139 kB 00:00 2026-02-24T16:38:47.742 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:47.742 INFO:teuthology.orchestra.run.trial052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:47.742 INFO:teuthology.orchestra.run.trial052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:38:47.742 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:47.742 INFO:teuthology.orchestra.run.trial052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:47.743 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:47.744 INFO:teuthology.orchestra.run.trial183.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 32 MB/s | 2.0 MB 00:00 2026-02-24T16:38:47.748 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:56 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:47.769 INFO:teuthology.orchestra.run.trial183.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.9 MB/s | 56 kB 00:00 2026-02-24T16:38:47.773 INFO:teuthology.orchestra.run.trial183.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 373 kB/s | 11 kB 00:00 2026-02-24T16:38:47.798 INFO:teuthology.orchestra.run.trial183.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 616 kB/s | 18 kB 00:00 2026-02-24T16:38:47.802 INFO:teuthology.orchestra.run.trial183.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 801 kB/s | 23 kB 00:00 2026-02-24T16:38:47.813 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:47.826 INFO:teuthology.orchestra.run.trial183.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 689 kB/s | 20 kB 00:00 2026-02-24T16:38:47.831 INFO:teuthology.orchestra.run.trial183.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 675 kB/s | 19 kB 00:00 2026-02-24T16:38:47.832 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:47.833 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:47.833 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:47.839 INFO:teuthology.orchestra.run.trial183.stdout:(79/96): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2026-02-24T16:38:47.854 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-osd 2026-02-24T16:38:47.855 INFO:teuthology.orchestra.run.trial183.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 923 kB/s | 26 kB 00:00 2026-02-24T16:38:47.869 INFO:teuthology.orchestra.run.trial183.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 27 MB/s | 1.0 MB 00:00 2026-02-24T16:38:47.870 INFO:teuthology.orchestra.run.trial183.stdout:(82/96): python3-more-itertools-8.12.0-2.el9.no 2.5 MB/s | 79 kB 00:00 2026-02-24T16:38:47.885 INFO:teuthology.orchestra.run.trial183.stdout:(83/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.8 MB/s | 86 kB 00:00 2026-02-24T16:38:47.898 INFO:teuthology.orchestra.run.trial183.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.0 MB/s | 58 kB 00:00 2026-02-24T16:38:47.899 INFO:teuthology.orchestra.run.trial183.stdout:(85/96): python3-portend-3.1.0-2.el9.noarch.rpm 568 kB/s | 16 kB 00:00 2026-02-24T16:38:47.903 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:47.903 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:47.904 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:47.904 INFO:teuthology.orchestra.run.trial153.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:47.904 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:47.904 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:47.915 INFO:teuthology.orchestra.run.trial183.stdout:(86/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.0 MB/s | 90 kB 00:00 2026-02-24T16:38:47.927 INFO:teuthology.orchestra.run.trial183.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-24T16:38:47.931 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-nbd 2026-02-24T16:38:47.944 INFO:teuthology.orchestra.run.trial183.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 2.0 MB/s | 59 kB 00:00 2026-02-24T16:38:47.955 INFO:teuthology.orchestra.run.trial183.stdout:(89/96): python3-routes-2.5.1-5.el9.noarch.rpm 3.3 MB/s | 188 kB 00:00 2026-02-24T16:38:47.957 INFO:teuthology.orchestra.run.trial183.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.1 MB/s | 125 kB 00:00 2026-02-24T16:38:47.974 INFO:teuthology.orchestra.run.trial183.stdout:(91/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.2 MB/s | 36 kB 00:00 2026-02-24T16:38:47.977 INFO:teuthology.orchestra.run.trial151.stdout:(18/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 73 kB/s | 40 kB 00:00 2026-02-24T16:38:47.984 INFO:teuthology.orchestra.run.trial183.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 2.9 MB/s | 86 kB 00:00 2026-02-24T16:38:47.990 INFO:teuthology.orchestra.run.trial183.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 2.7 MB/s | 90 kB 00:00 2026-02-24T16:38:48.004 INFO:teuthology.orchestra.run.trial183.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.6 MB/s | 48 kB 00:00 2026-02-24T16:38:48.015 INFO:teuthology.orchestra.run.trial183.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 719 kB/s | 22 kB 00:00 2026-02-24T16:38:48.019 INFO:teuthology.orchestra.run.trial183.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 687 kB/s | 20 kB 00:00 2026-02-24T16:38:48.020 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:48.020 INFO:teuthology.orchestra.run.trial183.stdout:Total 14 MB/s | 123 MB 00:08 2026-02-24T16:38:48.065 INFO:teuthology.orchestra.run.trial151.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 114 kB/s | 72 kB 00:00 2026-02-24T16:38:48.076 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.077 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.077 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:48.077 INFO:teuthology.orchestra.run.trial052.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:48.077 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.077 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:48.106 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install rbd-nbd 2026-02-24T16:38:48.172 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:57 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:48.237 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:48.256 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:48.256 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:49 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:48.257 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:48.257 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:48.278 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-osd-classic 2026-02-24T16:38:48.339 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:48.339 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:48.340 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 172 k 2026-02-24T16:38:48.341 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 462 k 2026-02-24T16:38:48.341 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-24T16:38:48.378 INFO:teuthology.orchestra.run.trial151.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 590 kB/s | 184 kB 00:00 2026-02-24T16:38:48.425 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:58 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:48.425 INFO:teuthology.orchestra.run.trial153.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2026-02-24T16:38:48.426 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:48.427 INFO:teuthology.orchestra.run.trial153.stdout:Total 1.9 MB/s | 172 kB 00:00 2026-02-24T16:38:48.427 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-24T16:38:48.431 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-24T16:38:48.431 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-24T16:38:48.447 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:38:48.463 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-24T16:38:48.463 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-24T16:38:48.486 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:38:48.486 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:38:48.509 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:48.510 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:48.510 INFO:teuthology.orchestra.run.trial052.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:48.510 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:48.510 INFO:teuthology.orchestra.run.trial052.stdout:Installing: 2026-02-24T16:38:48.510 INFO:teuthology.orchestra.run.trial052.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:Transaction Summary 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:================================================================================ 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:Install 1 Package 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:Total download size: 172 k 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:Installed size: 462 k 2026-02-24T16:38:48.511 INFO:teuthology.orchestra.run.trial052.stdout:Downloading Packages: 2026-02-24T16:38:48.520 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-24T16:38:48.537 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.591 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:57 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:48.597 INFO:teuthology.orchestra.run.trial052.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.9 MB/s | 172 kB 00:00 2026-02-24T16:38:48.598 INFO:teuthology.orchestra.run.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:48.598 INFO:teuthology.orchestra.run.trial052.stdout:Total 1.9 MB/s | 172 kB 00:00 2026-02-24T16:38:48.598 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction check 2026-02-24T16:38:48.603 INFO:teuthology.orchestra.run.trial052.stdout:Transaction check succeeded. 2026-02-24T16:38:48.603 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction test 2026-02-24T16:38:48.623 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.637 INFO:teuthology.orchestra.run.trial052.stdout:Transaction test succeeded. 2026-02-24T16:38:48.638 INFO:teuthology.orchestra.run.trial052.stdout:Running transaction 2026-02-24T16:38:48.656 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:48.675 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:48.676 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:48.676 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:48.687 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.687 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:48.688 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-24T16:38:48.688 INFO:teuthology.orchestra.run.trial153.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:48.688 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:38:48.688 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:48.694 INFO:teuthology.orchestra.run.trial052.stdout: Preparing : 1/1 2026-02-24T16:38:48.696 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-fuse 2026-02-24T16:38:48.712 INFO:teuthology.orchestra.run.trial052.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.716 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install nvme-cli 2026-02-24T16:38:48.769 INFO:teuthology.orchestra.run.trial151.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 648 kB/s | 253 kB 00:00 2026-02-24T16:38:48.801 INFO:teuthology.orchestra.run.trial052.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout:Installed: 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout: 2026-02-24T16:38:48.865 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:48.895 DEBUG:teuthology.orchestra.run.trial052:> sudo yum -y install nvme-cli 2026-02-24T16:38:48.981 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:38:48.981 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:38:49.008 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:58 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:49.031 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:50 ago on Tue 24 Feb 2026 04:37:59 PM UTC. 2026-02-24T16:38:49.076 INFO:teuthology.orchestra.run.trial151.stdout:(22/96): libgfortran-11.5.0-14.el9.x86_64.rpm 723 kB/s | 794 kB 00:01 2026-02-24T16:38:49.079 INFO:teuthology.orchestra.run.trial151.stdout:(23/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 211 kB/s | 351 kB 00:01 2026-02-24T16:38:49.092 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:49.092 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:49.092 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:49.093 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:49.094 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 987 k 2026-02-24T16:38:49.094 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 2.9 M 2026-02-24T16:38:49.094 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:49.096 INFO:teuthology.orchestra.run.trial153.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:38:49.114 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-24T16:38:49.114 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-24T16:38:49.114 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-24T16:38:49.137 DEBUG:teuthology.parallel:result is None 2026-02-24T16:38:49.213 INFO:teuthology.orchestra.run.trial051.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 8.0 MB/s | 987 kB 00:00 2026-02-24T16:38:49.213 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:49.213 INFO:teuthology.orchestra.run.trial051.stdout:Total 7.9 MB/s | 987 kB 00:00 2026-02-24T16:38:49.213 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:49.215 INFO:teuthology.orchestra.run.trial052.stdout:Last metadata expiration check: 0:00:59 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:38:49.218 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:49.218 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:49.254 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:49.255 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:49.283 INFO:teuthology.orchestra.run.trial052.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:38:49.300 INFO:teuthology.orchestra.run.trial052.stdout:Dependencies resolved. 2026-02-24T16:38:49.301 INFO:teuthology.orchestra.run.trial052.stdout:Nothing to do. 2026-02-24T16:38:49.301 INFO:teuthology.orchestra.run.trial052.stdout:Complete! 2026-02-24T16:38:49.313 INFO:teuthology.orchestra.run.trial151.stdout:(24/96): python3-ply-3.11-14.el9.noarch.rpm 449 kB/s | 106 kB 00:00 2026-02-24T16:38:49.324 DEBUG:teuthology.parallel:result is None 2026-02-24T16:38:49.346 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:49.364 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:49.474 INFO:teuthology.orchestra.run.trial151.stdout:(25/96): python3-pycparser-2.20-6.el9.noarch.rp 343 kB/s | 135 kB 00:00 2026-02-24T16:38:49.505 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:38:49.507 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:38:49.511 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:38:49.533 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:38:49.566 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:38:49.572 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:38:49.574 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:38:49.578 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:38:49.588 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:49.602 INFO:teuthology.orchestra.run.trial183.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:38:49.614 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:38:49.617 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:38:49.623 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:38:49.625 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:38:49.629 INFO:teuthology.orchestra.run.trial151.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 477 kB/s | 150 kB 00:00 2026-02-24T16:38:49.636 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:38:49.642 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:49.652 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:49.666 INFO:teuthology.orchestra.run.trial183.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:38:49.670 INFO:teuthology.orchestra.run.trial183.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:38:49.680 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-volume 2026-02-24T16:38:49.720 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:38:49.733 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:38:49.735 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:38:49.740 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:38:49.745 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:38:49.750 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:38:49.753 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:38:49.767 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:38:49.772 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:38:49.777 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:38:49.788 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:38:49.790 INFO:teuthology.orchestra.run.trial151.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 400 kB/s | 126 kB 00:00 2026-02-24T16:38:49.797 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:38:49.803 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:38:49.823 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:38:49.834 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:38:49.850 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:38:49.854 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:38:49.862 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:38:49.902 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:38:49.995 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:59 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:50.023 INFO:teuthology.orchestra.run.trial151.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 553 kB/s | 218 kB 00:00 2026-02-24T16:38:50.061 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:50.080 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:50.080 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:50.080 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:50.102 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados-devel 2026-02-24T16:38:50.184 INFO:teuthology.orchestra.run.trial151.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 461 kB/s | 182 kB 00:00 2026-02-24T16:38:50.226 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:38:50.236 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:38:50.240 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:38:50.246 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:38:50.247 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:38:50.275 INFO:teuthology.orchestra.run.trial183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:38:50.308 INFO:teuthology.orchestra.run.trial151.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 238 kB/s | 30 kB 00:00 2026-02-24T16:38:50.318 INFO:teuthology.orchestra.run.trial183.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:38:50.324 INFO:teuthology.orchestra.run.trial151.stdout:(31/96): python3-cryptography-36.0.1-5.el9.x86_ 820 kB/s | 1.2 MB 00:01 2026-02-24T16:38:50.327 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:38:50.332 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:38:50.336 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:38:50.341 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:38:50.345 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:38:50.352 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:38:50.355 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:38:50.383 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:38:50.393 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:38:50.401 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:38:50.407 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:38:50.417 INFO:teuthology.orchestra.run.trial151.stdout:(32/96): zip-3.0-35.el9.x86_64.rpm 675 kB/s | 266 kB 00:00 2026-02-24T16:38:50.418 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:59 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:50.441 INFO:teuthology.orchestra.run.trial151.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 126 kB/s | 15 kB 00:00 2026-02-24T16:38:50.442 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:38:50.502 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 123 k 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 437 k 2026-02-24T16:38:50.503 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:50.667 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:38:50.690 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:38:50.694 INFO:teuthology.orchestra.run.trial151.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 744 kB/s | 188 kB 00:00 2026-02-24T16:38:50.695 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:38:50.751 INFO:teuthology.orchestra.run.trial151.stdout:(35/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 737 kB/s | 246 kB 00:00 2026-02-24T16:38:50.752 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:38:50.754 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:38:50.775 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:38:50.793 INFO:teuthology.orchestra.run.trial151.stdout:(36/96): openblas-0.3.29-1.el9.x86_64.rpm 426 kB/s | 42 kB 00:00 2026-02-24T16:38:50.861 INFO:teuthology.orchestra.run.trial051.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 344 kB/s | 123 kB 00:00 2026-02-24T16:38:50.862 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:50.862 INFO:teuthology.orchestra.run.trial051.stdout:Total 343 kB/s | 123 kB 00:00 2026-02-24T16:38:50.862 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:50.864 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:50.864 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:50.897 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:50.897 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:50.952 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:50.970 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:51.057 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:51.099 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:51.120 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:51.147 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install libcephfs2 2026-02-24T16:38:51.172 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:38:51.466 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:00 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:51.531 INFO:teuthology.orchestra.run.trial051.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:51.549 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:51.550 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:51.550 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:51.571 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install libcephfs-devel 2026-02-24T16:38:51.844 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:38:51.861 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:38:51.871 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:38:51.877 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:38:51.887 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:00 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:51.891 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:38:51.907 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:38:51.972 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:51.972 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:51.972 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 37 k 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 171 k 2026-02-24T16:38:51.973 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:51.982 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:38:51.991 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:38:52.014 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:38:52.044 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:38:52.094 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:38:52.100 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:38:52.104 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:38:52.105 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:38:52.124 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:52.124 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:38:52.124 INFO:teuthology.orchestra.run.trial183.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:38:52.124 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:52.131 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:52.156 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:38:52.156 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:38:52.156 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:52.190 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:38:52.232 INFO:teuthology.orchestra.run.trial051.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 144 kB/s | 37 kB 00:00 2026-02-24T16:38:52.232 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:52.232 INFO:teuthology.orchestra.run.trial051.stdout:Total 143 kB/s | 37 kB 00:00 2026-02-24T16:38:52.232 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:52.235 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:52.235 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:52.238 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:52.238 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:52.260 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:52.277 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:52.339 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:52.402 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:52.402 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:52.402 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:52.403 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:52.403 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:52.403 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:52.430 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados2 2026-02-24T16:38:52.749 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:01 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:52.814 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:38:52.814 INFO:teuthology.orchestra.run.trial051.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:52.833 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:52.833 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:52.833 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:52.854 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librbd1 2026-02-24T16:38:53.168 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:02 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:53.233 INFO:teuthology.orchestra.run.trial051.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:53.252 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:53.253 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:53.253 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:53.274 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rados 2026-02-24T16:38:53.578 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:38:53.595 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:02 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:53.642 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:38:53.644 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:53.660 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:38:53.660 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:53.660 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:38:53.660 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:53.660 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:53.670 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:53.679 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:53.679 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:53.679 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:38:53.695 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:53.701 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rgw 2026-02-24T16:38:53.754 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:53.756 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:38:53.765 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:38:53.789 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:38:53.791 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:53.792 INFO:teuthology.orchestra.run.trial151.stdout:(37/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 878 kB/s | 3.0 MB 00:03 2026-02-24T16:38:54.018 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:54.084 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:54.103 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:54.103 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:54.103 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:54.124 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-cephfs 2026-02-24T16:38:54.129 INFO:teuthology.orchestra.run.trial151.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 727 kB/s | 244 kB 00:00 2026-02-24T16:38:54.441 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:54.465 INFO:teuthology.orchestra.run.trial151.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 740 kB/s | 249 kB 00:00 2026-02-24T16:38:54.506 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:54.525 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:54.526 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:54.526 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:54.548 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rbd 2026-02-24T16:38:54.565 INFO:teuthology.orchestra.run.trial151.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 475 kB/s | 48 kB 00:00 2026-02-24T16:38:54.823 INFO:teuthology.orchestra.run.trial151.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 687 kB/s | 177 kB 00:00 2026-02-24T16:38:54.864 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:54.930 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:54.948 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:54.949 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:54.949 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:54.970 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-fuse 2026-02-24T16:38:55.286 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:04 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:55.370 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 84 k 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 209 k 2026-02-24T16:38:55.371 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:55.393 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:38:55.400 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:55.512 INFO:teuthology.orchestra.run.trial051.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 598 kB/s | 84 kB 00:00 2026-02-24T16:38:55.514 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:55.514 INFO:teuthology.orchestra.run.trial051.stdout:Total 589 kB/s | 84 kB 00:00 2026-02-24T16:38:55.514 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:55.518 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:55.519 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:55.551 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:55.551 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:55.604 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:55.621 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:55.708 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:55.771 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:55.771 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:55.771 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:55.771 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:55.772 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:55.772 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:55.800 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-mirror 2026-02-24T16:38:55.858 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:38:55.859 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:55.873 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:38:55.874 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:55.937 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:38:55.979 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:38:55.980 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:56.002 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:38:56.002 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:56.002 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:38:56.003 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:56.003 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:38:56.003 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:56.016 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:56.118 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:05 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:56.200 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:56.201 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 2.9 M 2026-02-24T16:38:56.201 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 11 M 2026-02-24T16:38:56.201 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:56.245 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:38:56.246 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:56.261 INFO:teuthology.orchestra.run.trial151.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 871 kB/s | 1.2 MB 00:01 2026-02-24T16:38:56.268 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:38:56.268 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:56.268 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:38:56.268 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:56.268 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:38:56.269 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:56.361 INFO:teuthology.orchestra.run.trial151.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 348 kB/s | 35 kB 00:00 2026-02-24T16:38:56.363 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:38:56.383 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:56.384 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:56.391 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:56.416 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:38:56.467 INFO:teuthology.orchestra.run.trial051.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 11 MB/s | 2.9 MB 00:00 2026-02-24T16:38:56.468 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:56.469 INFO:teuthology.orchestra.run.trial051.stdout:Total 11 MB/s | 2.9 MB 00:00 2026-02-24T16:38:56.469 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:56.474 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:56.474 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:56.514 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:56.514 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:56.698 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:56.699 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:56.721 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:56.722 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:56.722 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:38:56.722 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:56.722 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:38:56.723 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:56.813 INFO:teuthology.orchestra.run.trial151.stdout:(44/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 893 kB/s | 5.3 MB 00:06 2026-02-24T16:38:57.042 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:57.043 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.043 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:57.043 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.043 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.043 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:57.070 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-nbd 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:38:57.168 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:38:57.169 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:38:57.170 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:38:57.171 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:38:57.172 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:38:57.173 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:38:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:38:57.175 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:38:57.176 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:38:57.176 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:38:57.239 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:38:57.239 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:57.239 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:38:57.239 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:38:57.239 INFO:teuthology.orchestra.run.trial183.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.240 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:38:57.241 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:38:57.242 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:38:57.243 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:38:57.244 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:38:57.245 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:38:57.246 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:57.247 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:57.299 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-base 2026-02-24T16:38:57.384 INFO:teuthology.orchestra.run.trial151.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 775 kB/s | 442 kB 00:00 2026-02-24T16:38:57.387 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:06 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:57.469 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:57.470 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:57.470 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:38:57.470 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:57.470 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-24T16:38:57.470 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 172 k 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 462 k 2026-02-24T16:38:57.471 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-24T16:38:57.566 INFO:teuthology.orchestra.run.trial151.stdout:(46/96): python3-babel-2.9.1-2.el9.noarch.rpm 901 kB/s | 6.0 MB 00:06 2026-02-24T16:38:57.567 INFO:teuthology.orchestra.run.trial151.stdout:(47/96): python3-packaging-20.9-5.el9.noarch.rp 421 kB/s | 77 kB 00:00 2026-02-24T16:38:57.623 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:44 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:38:57.643 INFO:teuthology.orchestra.run.trial051.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2026-02-24T16:38:57.643 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:57.643 INFO:teuthology.orchestra.run.trial051.stdout:Total 996 kB/s | 172 kB 00:00 2026-02-24T16:38:57.644 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-24T16:38:57.648 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-24T16:38:57.648 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-24T16:38:57.681 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-24T16:38:57.681 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-24T16:38:57.688 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:38:57.707 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:57.708 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:38:57.708 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:57.729 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install cephadm 2026-02-24T16:38:57.738 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-24T16:38:57.757 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:57.840 INFO:teuthology.orchestra.run.trial151.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 576 kB/s | 157 kB 00:00 2026-02-24T16:38:57.846 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:57.903 INFO:teuthology.orchestra.run.trial151.stdout:(49/96): python3-protobuf-3.14.0-17.el9.noarch. 794 kB/s | 267 kB 00:00 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-24T16:38:57.909 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:57.938 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install nvme-cli 2026-02-24T16:38:58.003 INFO:teuthology.orchestra.run.trial151.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 537 kB/s | 54 kB 00:00 2026-02-24T16:38:58.044 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:38:58.109 INFO:teuthology.orchestra.run.trial183.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:38:58.128 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:58.129 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:38:58.129 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:58.150 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:38:58.172 INFO:teuthology.orchestra.run.trial151.stdout:(51/96): python3-pyasn1-modules-0.4.8-7.el9.noa 835 kB/s | 277 kB 00:00 2026-02-24T16:38:58.254 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:07 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:38:58.271 INFO:teuthology.orchestra.run.trial151.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 420 kB/s | 42 kB 00:00 2026-02-24T16:38:58.319 INFO:teuthology.orchestra.run.trial051.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:38:58.337 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-24T16:38:58.338 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-24T16:38:58.338 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-24T16:38:58.360 DEBUG:teuthology.parallel:result is None 2026-02-24T16:38:58.464 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:45 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:38:58.526 INFO:teuthology.orchestra.run.trial151.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 739 kB/s | 189 kB 00:00 2026-02-24T16:38:58.549 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 147 k 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 419 k 2026-02-24T16:38:58.550 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:38:58.702 INFO:teuthology.orchestra.run.trial151.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 513 kB/s | 90 kB 00:00 2026-02-24T16:38:58.729 INFO:teuthology.orchestra.run.trial151.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 849 kB/s | 22 kB 00:00 2026-02-24T16:38:58.782 INFO:teuthology.orchestra.run.trial151.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 16 MB/s | 862 kB 00:00 2026-02-24T16:38:59.073 INFO:teuthology.orchestra.run.trial151.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.9 MB/s | 551 kB 00:00 2026-02-24T16:38:59.103 INFO:teuthology.orchestra.run.trial151.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 648 kB/s | 19 kB 00:00 2026-02-24T16:38:59.136 INFO:teuthology.orchestra.run.trial151.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2026-02-24T16:38:59.169 INFO:teuthology.orchestra.run.trial151.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 16 MB/s | 548 kB 00:00 2026-02-24T16:38:59.199 INFO:teuthology.orchestra.run.trial151.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 998 kB/s | 29 kB 00:00 2026-02-24T16:38:59.231 INFO:teuthology.orchestra.run.trial151.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.8 MB/s | 60 kB 00:00 2026-02-24T16:38:59.260 INFO:teuthology.orchestra.run.trial151.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.5 MB/s | 43 kB 00:00 2026-02-24T16:38:59.266 INFO:teuthology.orchestra.run.trial183.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 205 kB/s | 147 kB 00:00 2026-02-24T16:38:59.266 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:38:59.266 INFO:teuthology.orchestra.run.trial183.stdout:Total 205 kB/s | 147 kB 00:00 2026-02-24T16:38:59.267 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:38:59.272 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:38:59.272 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:38:59.290 INFO:teuthology.orchestra.run.trial151.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.1 MB/s | 32 kB 00:00 2026-02-24T16:38:59.303 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:38:59.303 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:38:59.317 INFO:teuthology.orchestra.run.trial151.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 499 kB/s | 14 kB 00:00 2026-02-24T16:38:59.351 INFO:teuthology.orchestra.run.trial151.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.1 MB/s | 173 kB 00:00 2026-02-24T16:38:59.359 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:38:59.361 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:59.384 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:59.384 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:38:59.384 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:38:59.384 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:59.388 INFO:teuthology.orchestra.run.trial151.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 9.4 MB/s | 358 kB 00:00 2026-02-24T16:38:59.420 INFO:teuthology.orchestra.run.trial151.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 8.1 MB/s | 254 kB 00:00 2026-02-24T16:38:59.503 INFO:teuthology.orchestra.run.trial151.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 24 MB/s | 2.0 MB 00:00 2026-02-24T16:38:59.537 INFO:teuthology.orchestra.run.trial151.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.2 MB/s | 144 kB 00:00 2026-02-24T16:38:59.573 INFO:teuthology.orchestra.run.trial151.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.8 MB/s | 139 kB 00:00 2026-02-24T16:38:59.604 INFO:teuthology.orchestra.run.trial151.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.7 MB/s | 56 kB 00:00 2026-02-24T16:38:59.633 INFO:teuthology.orchestra.run.trial151.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 379 kB/s | 11 kB 00:00 2026-02-24T16:38:59.661 INFO:teuthology.orchestra.run.trial151.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 624 kB/s | 18 kB 00:00 2026-02-24T16:38:59.690 INFO:teuthology.orchestra.run.trial151.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 788 kB/s | 23 kB 00:00 2026-02-24T16:38:59.710 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:38:59.710 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:59.710 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:38:59.710 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:38:59.710 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:38:59.711 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:38:59.721 INFO:teuthology.orchestra.run.trial151.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 648 kB/s | 20 kB 00:00 2026-02-24T16:38:59.738 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr 2026-02-24T16:38:59.752 INFO:teuthology.orchestra.run.trial151.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 618 kB/s | 19 kB 00:00 2026-02-24T16:38:59.783 INFO:teuthology.orchestra.run.trial151.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 851 kB/s | 26 kB 00:00 2026-02-24T16:38:59.928 INFO:teuthology.orchestra.run.trial151.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2026-02-24T16:38:59.959 INFO:teuthology.orchestra.run.trial151.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.5 MB/s | 79 kB 00:00 2026-02-24T16:38:59.991 INFO:teuthology.orchestra.run.trial151.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.6 MB/s | 86 kB 00:00 2026-02-24T16:39:00.021 INFO:teuthology.orchestra.run.trial151.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.9 MB/s | 58 kB 00:00 2026-02-24T16:39:00.050 INFO:teuthology.orchestra.run.trial151.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 567 kB/s | 16 kB 00:00 2026-02-24T16:39:00.063 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:00.079 INFO:teuthology.orchestra.run.trial151.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.0 MB/s | 90 kB 00:00 2026-02-24T16:39:00.109 INFO:teuthology.orchestra.run.trial151.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-24T16:39:00.128 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:00.143 INFO:teuthology.orchestra.run.trial151.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2026-02-24T16:39:00.147 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:00.147 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:00.148 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:00.169 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:39:00.171 INFO:teuthology.orchestra.run.trial151.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 2.0 MB/s | 59 kB 00:00 2026-02-24T16:39:00.206 INFO:teuthology.orchestra.run.trial151.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.5 MB/s | 125 kB 00:00 2026-02-24T16:39:00.238 INFO:teuthology.orchestra.run.trial151.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.1 MB/s | 36 kB 00:00 2026-02-24T16:39:00.267 INFO:teuthology.orchestra.run.trial151.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 2.9 MB/s | 86 kB 00:00 2026-02-24T16:39:00.296 INFO:teuthology.orchestra.run.trial151.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 3.1 MB/s | 90 kB 00:00 2026-02-24T16:39:00.327 INFO:teuthology.orchestra.run.trial151.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.5 MB/s | 48 kB 00:00 2026-02-24T16:39:00.356 INFO:teuthology.orchestra.run.trial151.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 775 kB/s | 22 kB 00:00 2026-02-24T16:39:00.385 INFO:teuthology.orchestra.run.trial151.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 694 kB/s | 20 kB 00:00 2026-02-24T16:39:00.487 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:00.552 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:00.570 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:00.571 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:00.571 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:00.592 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:39:00.909 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:00.974 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:00.993 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:00.993 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:00.993 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:01.015 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:39:01.334 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:01.399 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:01.418 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:01.418 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:01.418 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:01.440 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:39:01.760 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:01.826 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:01.845 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:01.846 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:01.846 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:01.868 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd 2026-02-24T16:39:02.183 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:49 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:02.249 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:02.267 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:02.268 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:02.268 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:02.290 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-classic 2026-02-24T16:39:02.608 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:49 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:02.674 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:02.692 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:02.693 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:02.693 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:02.714 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-fuse 2026-02-24T16:39:03.030 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:50 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:03.115 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:03.116 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 987 k 2026-02-24T16:39:03.116 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 2.9 M 2026-02-24T16:39:03.116 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:03.237 INFO:teuthology.orchestra.run.trial183.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 8.0 MB/s | 987 kB 00:00 2026-02-24T16:39:03.237 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:03.237 INFO:teuthology.orchestra.run.trial183.stdout:Total 8.0 MB/s | 987 kB 00:00 2026-02-24T16:39:03.237 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:03.241 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:03.241 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:03.276 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:03.276 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:03.369 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:03.387 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:03.498 INFO:teuthology.orchestra.run.trial151.stdout:(95/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 879 kB/s | 6.1 MB 00:07 2026-02-24T16:39:03.614 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:03.676 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:03.677 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:03.677 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:03.677 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:03.677 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:03.677 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:03.707 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-volume 2026-02-24T16:39:04.030 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:51 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:04.097 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:04.116 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:04.116 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:04.116 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:04.138 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados-devel 2026-02-24T16:39:04.464 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:51 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:04.548 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 123 k 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 437 k 2026-02-24T16:39:04.549 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:04.661 INFO:teuthology.orchestra.run.trial183.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 1.1 MB/s | 123 kB 00:00 2026-02-24T16:39:04.661 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:04.661 INFO:teuthology.orchestra.run.trial183.stdout:Total 1.1 MB/s | 123 kB 00:00 2026-02-24T16:39:04.661 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:04.663 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:04.663 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:04.694 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:04.695 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:04.749 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:04.767 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:04.854 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:04.916 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:04.917 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:04.917 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:04.917 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:04.917 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:04.917 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:04.946 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs2 2026-02-24T16:39:05.266 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:05.331 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:05.350 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:05.350 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:05.351 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:05.372 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs-devel 2026-02-24T16:39:05.695 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:05.782 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 37 k 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 171 k 2026-02-24T16:39:05.783 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:05.847 INFO:teuthology.orchestra.run.trial183.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 589 kB/s | 37 kB 00:00 2026-02-24T16:39:05.848 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:05.848 INFO:teuthology.orchestra.run.trial183.stdout:Total 580 kB/s | 37 kB 00:00 2026-02-24T16:39:05.848 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:05.850 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:05.850 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:05.852 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:05.853 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:05.875 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:05.892 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:05.956 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:06.018 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:06.018 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:06.019 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:06.019 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:06.019 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:06.019 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:06.046 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados2 2026-02-24T16:39:06.367 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:06.431 INFO:teuthology.orchestra.run.trial183.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:06.450 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:06.450 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:06.450 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:06.472 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librbd1 2026-02-24T16:39:06.799 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:53 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:06.864 INFO:teuthology.orchestra.run.trial183.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:06.883 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:06.884 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:06.884 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:06.905 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rados 2026-02-24T16:39:07.228 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:54 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:07.294 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:07.312 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:07.313 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:07.314 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:07.336 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rgw 2026-02-24T16:39:07.660 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:54 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:07.726 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:07.744 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:07.745 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:07.745 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:07.768 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-cephfs 2026-02-24T16:39:08.087 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:08.155 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:08.174 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:08.175 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:08.175 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:08.196 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rbd 2026-02-24T16:39:08.518 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:08.587 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:08.605 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:08.606 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:08.606 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:08.627 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-fuse 2026-02-24T16:39:08.941 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:09.026 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:09.026 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.027 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 84 k 2026-02-24T16:39:09.028 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 209 k 2026-02-24T16:39:09.028 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:09.195 INFO:teuthology.orchestra.run.trial183.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 501 kB/s | 84 kB 00:00 2026-02-24T16:39:09.195 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:09.195 INFO:teuthology.orchestra.run.trial183.stdout:Total 498 kB/s | 84 kB 00:00 2026-02-24T16:39:09.196 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:09.199 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:09.200 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:09.232 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:09.232 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:09.284 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:09.301 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:09.392 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:09.456 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:09.457 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.457 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:09.457 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:09.457 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.458 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:09.485 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-mirror 2026-02-24T16:39:09.802 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:56 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:09.886 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 2.9 M 2026-02-24T16:39:09.887 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 11 M 2026-02-24T16:39:09.887 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:10.033 INFO:teuthology.orchestra.run.trial183.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 20 MB/s | 2.9 MB 00:00 2026-02-24T16:39:10.034 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:10.034 INFO:teuthology.orchestra.run.trial183.stdout:Total 20 MB/s | 2.9 MB 00:00 2026-02-24T16:39:10.034 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:10.039 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:10.039 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:10.079 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:10.079 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:10.257 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:10.259 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:10.281 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:10.282 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:10.282 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:39:10.282 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:39:10.282 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:39:10.282 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:10.607 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:10.607 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:10.607 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:10.608 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:10.608 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:10.608 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:10.637 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-nbd 2026-02-24T16:39:10.959 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:57 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:11.042 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-24T16:39:11.043 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:11.044 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 172 k 2026-02-24T16:39:11.044 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 462 k 2026-02-24T16:39:11.044 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-24T16:39:11.130 INFO:teuthology.orchestra.run.trial183.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.9 MB/s | 172 kB 00:00 2026-02-24T16:39:11.131 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:11.131 INFO:teuthology.orchestra.run.trial183.stdout:Total 1.9 MB/s | 172 kB 00:00 2026-02-24T16:39:11.131 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-24T16:39:11.135 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-24T16:39:11.135 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-24T16:39:11.169 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-24T16:39:11.169 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-24T16:39:11.225 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-24T16:39:11.242 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:11.333 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:39:11.396 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:11.425 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install nvme-cli 2026-02-24T16:39:11.745 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:58 ago on Tue 24 Feb 2026 04:38:13 PM UTC. 2026-02-24T16:39:11.810 INFO:teuthology.orchestra.run.trial183.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:39:11.827 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-24T16:39:11.828 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-24T16:39:11.828 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-24T16:39:11.853 DEBUG:teuthology.parallel:result is None 2026-02-24T16:39:20.071 INFO:teuthology.orchestra.run.trial151.stdout:(96/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 894 kB/s | 19 MB 00:22 2026-02-24T16:39:20.073 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:20.073 INFO:teuthology.orchestra.run.trial151.stdout:Total 3.4 MB/s | 123 MB 00:36 2026-02-24T16:39:20.465 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:20.504 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:20.504 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:21.004 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:21.004 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:21.534 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:21.536 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-24T16:39:21.540 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-24T16:39:21.561 INFO:teuthology.orchestra.run.trial151.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-24T16:39:21.594 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-24T16:39:21.600 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-24T16:39:21.603 INFO:teuthology.orchestra.run.trial151.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-24T16:39:21.607 INFO:teuthology.orchestra.run.trial151.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-24T16:39:21.630 INFO:teuthology.orchestra.run.trial151.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-24T16:39:21.643 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-24T16:39:21.647 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-24T16:39:21.653 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-24T16:39:21.654 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-24T16:39:21.666 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-24T16:39:21.672 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-24T16:39:21.696 INFO:teuthology.orchestra.run.trial151.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-24T16:39:21.699 INFO:teuthology.orchestra.run.trial151.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-24T16:39:21.749 INFO:teuthology.orchestra.run.trial151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-24T16:39:21.762 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-24T16:39:21.764 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-24T16:39:21.768 INFO:teuthology.orchestra.run.trial151.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-24T16:39:21.774 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-24T16:39:21.778 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-24T16:39:21.781 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-24T16:39:21.796 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-24T16:39:21.800 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-24T16:39:21.805 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-24T16:39:21.816 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-24T16:39:21.825 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-24T16:39:21.831 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-24T16:39:21.851 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-24T16:39:21.863 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-24T16:39:21.878 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-24T16:39:21.882 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-24T16:39:21.890 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-24T16:39:21.930 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-24T16:39:22.255 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-24T16:39:22.265 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-24T16:39:22.269 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-24T16:39:22.275 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-24T16:39:22.276 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-24T16:39:22.304 INFO:teuthology.orchestra.run.trial151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-24T16:39:22.349 INFO:teuthology.orchestra.run.trial151.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-24T16:39:22.359 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-24T16:39:22.364 INFO:teuthology.orchestra.run.trial151.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-24T16:39:22.368 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-24T16:39:22.373 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-24T16:39:22.377 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-24T16:39:22.384 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-24T16:39:22.387 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-24T16:39:22.415 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-24T16:39:22.425 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-24T16:39:22.431 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-24T16:39:22.438 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-24T16:39:22.472 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-24T16:39:22.698 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-24T16:39:22.722 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-24T16:39:22.725 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-24T16:39:22.784 INFO:teuthology.orchestra.run.trial151.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-24T16:39:22.785 INFO:teuthology.orchestra.run.trial151.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-24T16:39:22.806 INFO:teuthology.orchestra.run.trial151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-24T16:39:23.133 INFO:teuthology.orchestra.run.trial151.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-24T16:39:23.206 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-24T16:39:23.881 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-24T16:39:23.898 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-24T16:39:23.908 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-24T16:39:23.915 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-24T16:39:23.929 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-24T16:39:23.945 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-24T16:39:24.021 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-24T16:39:24.029 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-24T16:39:24.053 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-24T16:39:24.083 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-24T16:39:24.134 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-24T16:39:24.139 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-24T16:39:24.143 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-24T16:39:24.144 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-24T16:39:24.161 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:39:24.161 INFO:teuthology.orchestra.run.trial151.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:39:24.161 INFO:teuthology.orchestra.run.trial151.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:39:24.161 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:24.168 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:39:24.193 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-24T16:39:24.193 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:39:24.193 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:24.226 INFO:teuthology.orchestra.run.trial151.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-24T16:39:24.845 INFO:teuthology.orchestra.run.trial151.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-24T16:39:25.608 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 80/96 2026-02-24T16:39:25.674 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 81/96 2026-02-24T16:39:25.674 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:39:25.688 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 82/96 2026-02-24T16:39:25.688 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:25.688 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:39:25.689 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:25.698 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:39:25.722 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 83/96 2026-02-24T16:39:25.722 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:25.722 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:39:25.723 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:39:25.723 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:39:25.723 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:25.780 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:39:25.781 INFO:teuthology.orchestra.run.trial151.stdout: Installing : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 84/96 2026-02-24T16:39:25.790 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 85/96 2026-02-24T16:39:25.814 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 86/96 2026-02-24T16:39:25.816 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:39:27.403 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 87/96 2026-02-24T16:39:27.410 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:39:27.867 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 88/96 2026-02-24T16:39:27.868 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:39:27.880 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 89/96 2026-02-24T16:39:27.880 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:39:27.941 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 90/96 2026-02-24T16:39:27.983 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 91/96 2026-02-24T16:39:27.984 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:39:28.005 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 92/96 2026-02-24T16:39:28.006 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:28.006 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:39:28.006 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:39:28.006 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:39:28.006 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:28.017 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:39:28.240 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 93/96 2026-02-24T16:39:28.242 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:39:28.263 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 94/96 2026-02-24T16:39:28.264 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:28.264 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:39:28.264 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:39:28.264 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:39:28.264 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:28.358 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 95/96 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:39:28.378 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:28.379 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:39:28.386 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:39:28.410 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 96/96 2026-02-24T16:39:29.194 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 96/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-2:20.3.0-5507.g618c318a.el9.x86_64 1/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 3/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 4/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 5/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_6 6/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_6 7/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el 8/96 2026-02-24T16:39:29.195 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarc 9/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noa 10/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c3 11/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noa 12/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9. 13/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 14/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9 15/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 16/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : cephadm-2:20.3.0-5507.g618c318a.el9.noarch 17/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-24T16:39:29.196 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-24T16:39:29.197 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-24T16:39:29.198 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-24T16:39:29.199 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-24T16:39:29.200 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-24T16:39:29.201 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-24T16:39:29.202 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-24T16:39:29.203 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-24T16:39:29.203 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-grafana-dashboards-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mds-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-k8sevents-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-modules-core-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-mon-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-osd-crimson-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-prometheus-alerts-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: cephadm-2:20.3.0-5507.g618c318a.el9.noarch 2026-02-24T16:39:29.264 INFO:teuthology.orchestra.run.trial151.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:39:29.265 INFO:teuthology.orchestra.run.trial151.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:29.266 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:29.321 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-base 2026-02-24T16:39:29.643 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:17 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:29.708 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-base-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:29.727 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:29.727 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:29.727 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:29.749 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install cephadm 2026-02-24T16:39:30.065 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:18 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:30.130 INFO:teuthology.orchestra.run.trial151.stdout:Package cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:30.149 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:30.149 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:30.149 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:30.170 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:39:30.486 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:18 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repo Size 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:30.569 INFO:teuthology.orchestra.run.trial151.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5507.g618c318a.el9 ceph 147 k 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 147 k 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 419 k 2026-02-24T16:39:30.570 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:30.938 INFO:teuthology.orchestra.run.trial151.stdout:ceph-immutable-object-cache-20.3.0-5507.g618c31 400 kB/s | 147 kB 00:00 2026-02-24T16:39:30.939 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:30.940 INFO:teuthology.orchestra.run.trial151.stdout:Total 399 kB/s | 147 kB 00:00 2026-02-24T16:39:30.940 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:30.944 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:30.944 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:30.975 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:30.976 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:31.031 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:31.032 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:39:31.054 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:39:31.054 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:31.054 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:39:31.054 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:31.382 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5507.g618c318a. 1/1 2026-02-24T16:39:31.382 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:31.383 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:31.383 INFO:teuthology.orchestra.run.trial151.stdout: ceph-immutable-object-cache-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:31.383 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:31.383 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:31.410 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-mgr 2026-02-24T16:39:31.730 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:19 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:31.796 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-mgr-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:31.814 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:31.815 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:31.815 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:31.836 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:39:32.151 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:20 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:32.216 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-mgr-dashboard-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:32.235 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:32.235 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:32.236 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:32.258 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:39:32.567 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:20 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:32.632 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:32.651 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:32.652 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:32.652 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:32.672 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:39:32.986 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:20 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:33.053 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-mgr-rook-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:33.072 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:33.072 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:33.072 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:33.093 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:39:33.410 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:21 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:33.476 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-mgr-cephadm-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:33.495 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:33.496 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:33.496 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:33.517 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-osd 2026-02-24T16:39:33.830 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:21 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:33.895 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-osd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:33.914 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:33.915 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:33.915 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:33.937 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-osd-classic 2026-02-24T16:39:34.259 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:22 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:34.325 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-osd-classic-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:34.344 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:34.345 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:34.345 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:34.368 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-fuse 2026-02-24T16:39:34.689 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:22 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:34.774 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:34.774 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:34.774 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repo Size 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout: ceph-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 987 k 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 987 k 2026-02-24T16:39:34.775 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 2.9 M 2026-02-24T16:39:34.776 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:35.288 INFO:teuthology.orchestra.run.trial151.stdout:ceph-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.9 MB/s | 987 kB 00:00 2026-02-24T16:39:35.288 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:35.289 INFO:teuthology.orchestra.run.trial151.stdout:Total 1.9 MB/s | 987 kB 00:00 2026-02-24T16:39:35.289 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:35.294 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:35.294 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:35.329 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:35.329 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:35.421 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:35.439 INFO:teuthology.orchestra.run.trial151.stdout: Installing : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:35.687 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:35.749 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:35.749 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:35.749 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:35.749 INFO:teuthology.orchestra.run.trial151.stdout: ceph-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:35.749 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:35.750 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:35.777 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install ceph-volume 2026-02-24T16:39:36.097 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:24 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:36.163 INFO:teuthology.orchestra.run.trial151.stdout:Package ceph-volume-2:20.3.0-5507.g618c318a.el9.noarch is already installed. 2026-02-24T16:39:36.182 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:36.183 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:36.183 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:36.204 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install librados-devel 2026-02-24T16:39:36.519 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:24 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repo Size 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout: librados-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 123 k 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:36.607 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 123 k 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 437 k 2026-02-24T16:39:36.608 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:36.693 INFO:teuthology.orchestra.run.trial151.stdout:librados-devel-20.3.0-5507.g618c318a.el9.x86_64 1.4 MB/s | 123 kB 00:00 2026-02-24T16:39:36.694 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:36.694 INFO:teuthology.orchestra.run.trial151.stdout:Total 1.4 MB/s | 123 kB 00:00 2026-02-24T16:39:36.694 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:36.697 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:36.697 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:36.728 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:36.728 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:36.783 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:36.801 INFO:teuthology.orchestra.run.trial151.stdout: Installing : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:36.887 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:36.952 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:36.953 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:36.953 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:36.953 INFO:teuthology.orchestra.run.trial151.stdout: librados-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:36.953 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:36.953 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:36.984 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install libcephfs2 2026-02-24T16:39:37.308 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:25 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:37.374 INFO:teuthology.orchestra.run.trial151.stdout:Package libcephfs2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:37.393 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:37.393 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:37.393 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:37.415 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install libcephfs-devel 2026-02-24T16:39:37.729 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:25 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repo Size 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-devel x86_64 2:20.3.0-5507.g618c318a.el9 ceph 37 k 2026-02-24T16:39:37.815 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 37 k 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 171 k 2026-02-24T16:39:37.816 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:37.887 INFO:teuthology.orchestra.run.trial151.stdout:libcephfs-devel-20.3.0-5507.g618c318a.el9.x86_6 516 kB/s | 37 kB 00:00 2026-02-24T16:39:37.888 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:37.888 INFO:teuthology.orchestra.run.trial151.stdout:Total 507 kB/s | 37 kB 00:00 2026-02-24T16:39:37.888 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:37.890 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:37.890 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:37.892 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:37.893 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:37.915 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:37.932 INFO:teuthology.orchestra.run.trial151.stdout: Installing : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:37.992 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:38.054 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:38.054 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:38.054 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:38.054 INFO:teuthology.orchestra.run.trial151.stdout: libcephfs-devel-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:38.055 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:38.055 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:38.082 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install librados2 2026-02-24T16:39:38.401 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:26 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:38.466 INFO:teuthology.orchestra.run.trial151.stdout:Package librados2-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:38.485 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:38.486 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:38.486 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:38.508 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install librbd1 2026-02-24T16:39:38.823 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:26 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:38.889 INFO:teuthology.orchestra.run.trial151.stdout:Package librbd1-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:38.908 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:38.909 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:38.909 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:38.930 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install python3-rados 2026-02-24T16:39:39.245 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:27 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:39.310 INFO:teuthology.orchestra.run.trial151.stdout:Package python3-rados-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:39.328 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:39.329 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:39.329 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:39.350 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install python3-rgw 2026-02-24T16:39:39.670 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:27 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:39.735 INFO:teuthology.orchestra.run.trial151.stdout:Package python3-rgw-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:39.754 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:39.755 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:39.755 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:39.777 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install python3-cephfs 2026-02-24T16:39:40.097 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:40.162 INFO:teuthology.orchestra.run.trial151.stdout:Package python3-cephfs-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:40.181 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:40.182 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:40.182 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:40.203 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install python3-rbd 2026-02-24T16:39:40.519 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:40.585 INFO:teuthology.orchestra.run.trial151.stdout:Package python3-rbd-2:20.3.0-5507.g618c318a.el9.x86_64 is already installed. 2026-02-24T16:39:40.604 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:40.604 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:40.604 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:40.625 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install rbd-fuse 2026-02-24T16:39:40.941 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:41.028 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:41.028 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.028 INFO:teuthology.orchestra.run.trial151.stdout: Package Architecture Version Repository Size 2026-02-24T16:39:41.028 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout: rbd-fuse x86_64 2:20.3.0-5507.g618c318a.el9 ceph 84 k 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 84 k 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 209 k 2026-02-24T16:39:41.029 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:41.102 INFO:teuthology.orchestra.run.trial151.stdout:rbd-fuse-20.3.0-5507.g618c318a.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2026-02-24T16:39:41.103 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:41.103 INFO:teuthology.orchestra.run.trial151.stdout:Total 1.1 MB/s | 84 kB 00:00 2026-02-24T16:39:41.103 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:41.106 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:41.107 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:41.139 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:41.139 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:41.190 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:41.208 INFO:teuthology.orchestra.run.trial151.stdout: Installing : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:41.302 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout: rbd-fuse-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.364 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:41.393 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install rbd-mirror 2026-02-24T16:39:41.711 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:29 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout: Package Arch Version Repo Size 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:41.794 INFO:teuthology.orchestra.run.trial151.stdout: rbd-mirror x86_64 2:20.3.0-5507.g618c318a.el9 ceph 2.9 M 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 2.9 M 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 11 M 2026-02-24T16:39:41.795 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:41.941 INFO:teuthology.orchestra.run.trial151.stdout:rbd-mirror-20.3.0-5507.g618c318a.el9.x86_64.rpm 20 MB/s | 2.9 MB 00:00 2026-02-24T16:39:41.942 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:41.942 INFO:teuthology.orchestra.run.trial151.stdout:Total 19 MB/s | 2.9 MB 00:00 2026-02-24T16:39:41.942 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:41.947 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:41.947 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:41.988 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:41.989 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:42.165 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:42.167 INFO:teuthology.orchestra.run.trial151.stdout: Installing : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:42.189 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:42.190 INFO:teuthology.orchestra.run.trial151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:39:42.190 INFO:teuthology.orchestra.run.trial151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:39:42.190 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:39:42.190 INFO:teuthology.orchestra.run.trial151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:39:42.191 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:42.492 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:42.493 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:42.493 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:42.493 INFO:teuthology.orchestra.run.trial151.stdout: rbd-mirror-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:42.493 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:42.493 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:42.522 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install rbd-nbd 2026-02-24T16:39:42.839 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:30 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:42.923 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:42.923 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout: Package Architecture Version Repository Size 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:Installing: 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout: rbd-nbd x86_64 2:20.3.0-5507.g618c318a.el9 ceph 172 k 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:Transaction Summary 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:================================================================================ 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:Install 1 Package 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:42.924 INFO:teuthology.orchestra.run.trial151.stdout:Total download size: 172 k 2026-02-24T16:39:42.925 INFO:teuthology.orchestra.run.trial151.stdout:Installed size: 462 k 2026-02-24T16:39:42.925 INFO:teuthology.orchestra.run.trial151.stdout:Downloading Packages: 2026-02-24T16:39:43.010 INFO:teuthology.orchestra.run.trial151.stdout:rbd-nbd-20.3.0-5507.g618c318a.el9.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2026-02-24T16:39:43.011 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:39:43.011 INFO:teuthology.orchestra.run.trial151.stdout:Total 1.9 MB/s | 172 kB 00:00 2026-02-24T16:39:43.011 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction check 2026-02-24T16:39:43.015 INFO:teuthology.orchestra.run.trial151.stdout:Transaction check succeeded. 2026-02-24T16:39:43.015 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction test 2026-02-24T16:39:43.049 INFO:teuthology.orchestra.run.trial151.stdout:Transaction test succeeded. 2026-02-24T16:39:43.049 INFO:teuthology.orchestra.run.trial151.stdout:Running transaction 2026-02-24T16:39:43.106 INFO:teuthology.orchestra.run.trial151.stdout: Preparing : 1/1 2026-02-24T16:39:43.123 INFO:teuthology.orchestra.run.trial151.stdout: Installing : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:43.210 INFO:teuthology.orchestra.run.trial151.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout: Verifying : rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 1/1 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout:Installed: 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout: rbd-nbd-2:20.3.0-5507.g618c318a.el9.x86_64 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:39:43.273 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:43.302 DEBUG:teuthology.orchestra.run.trial151:> sudo yum -y install nvme-cli 2026-02-24T16:39:43.634 INFO:teuthology.orchestra.run.trial151.stdout:Last metadata expiration check: 0:01:31 ago on Tue 24 Feb 2026 04:38:12 PM UTC. 2026-02-24T16:39:43.699 INFO:teuthology.orchestra.run.trial151.stdout:Package nvme-cli-2.16-1.el9.x86_64 is already installed. 2026-02-24T16:39:43.716 INFO:teuthology.orchestra.run.trial151.stdout:Dependencies resolved. 2026-02-24T16:39:43.717 INFO:teuthology.orchestra.run.trial151.stdout:Nothing to do. 2026-02-24T16:39:43.717 INFO:teuthology.orchestra.run.trial151.stdout:Complete! 2026-02-24T16:39:43.743 DEBUG:teuthology.parallel:result is None 2026-02-24T16:39:43.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:43.914 DEBUG:teuthology.orchestra.run.trial051:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:43.936 INFO:teuthology.orchestra.run.trial051.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:43.936 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:43.936 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:43.939 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:44.079 DEBUG:teuthology.orchestra.run.trial052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:44.100 INFO:teuthology.orchestra.run.trial052.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.101 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.101 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:44.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:44.256 DEBUG:teuthology.orchestra.run.trial151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:44.278 INFO:teuthology.orchestra.run.trial151.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.278 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.278 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:44.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:44.427 DEBUG:teuthology.orchestra.run.trial153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:44.449 INFO:teuthology.orchestra.run.trial153.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.449 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.449 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:44.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:44.610 DEBUG:teuthology.orchestra.run.trial183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:44.631 INFO:teuthology.orchestra.run.trial183.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.631 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.631 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:44.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:44.749 DEBUG:teuthology.orchestra.run.trial191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:39:44.780 INFO:teuthology.orchestra.run.trial191.stdout:20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.780 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507.g618c318a.el9 2026-02-24T16:39:44.781 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507.g618c318a is installed. 2026-02-24T16:39:44.782 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T16:39:44.783 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:39:44.783 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.807 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:39:44.807 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.834 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:39:44.834 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.862 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:39:44.862 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.889 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:39:44.889 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.915 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:39:44.915 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:39:44.942 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T16:39:44.942 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:39:44.942 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:44.968 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.032 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:39:45.033 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:45.058 DEBUG:teuthology.orchestra.run.trial052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.123 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:39:45.123 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:45.149 DEBUG:teuthology.orchestra.run.trial151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.214 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:39:45.214 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:45.240 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.304 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:39:45.304 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:45.330 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.393 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:39:45.394 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:39:45.420 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:39:45.484 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T16:39:45.484 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:39:45.484 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.510 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:45.572 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:39:45.573 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.597 DEBUG:teuthology.orchestra.run.trial052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:45.663 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:39:45.663 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.689 DEBUG:teuthology.orchestra.run.trial151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:45.753 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:39:45.753 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.777 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:45.842 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:39:45.842 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.867 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:45.931 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:39:45.931 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:39:45.956 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:39:46.020 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T16:39:46.020 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:39:46.020 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.044 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.107 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:39:46.107 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.133 DEBUG:teuthology.orchestra.run.trial052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.196 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:39:46.197 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.222 DEBUG:teuthology.orchestra.run.trial151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.286 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:39:46.286 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.312 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.374 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:39:46.375 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.399 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.463 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:39:46.463 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:39:46.489 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:39:46.553 INFO:teuthology.run_tasks:Running task cephadm... 2026-02-24T16:39:46.682 INFO:tasks.cephadm:Config: {'conf': {'global': {'mon warn on pool no app': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'OSD_DOWN', 'OSD_HOST_DOWN', 'OSD_ROOT_DOWN', 'PG_DEGRADED', 'NVMEOF_SINGLE_GATEWAY', 'CEPHADM_FAILED_DAEMON'], 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'} 2026-02-24T16:39:46.682 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:46.685 INFO:tasks.cephadm:Cluster fsid is 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:39:46.685 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2026-02-24T16:39:46.685 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '10.20.193.151', 'mon.b': '10.20.193.153', 'mon.c': '10.20.193.183'} 2026-02-24T16:39:46.685 INFO:tasks.cephadm:First mon is mon.a on trial151 2026-02-24T16:39:46.685 INFO:tasks.cephadm:First mgr is x 2026-02-24T16:39:46.685 INFO:tasks.cephadm:Normalizing hostnames... 2026-02-24T16:39:46.685 DEBUG:teuthology.orchestra.run.trial051:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.710 DEBUG:teuthology.orchestra.run.trial052:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.735 DEBUG:teuthology.orchestra.run.trial151:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.761 DEBUG:teuthology.orchestra.run.trial153:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.787 DEBUG:teuthology.orchestra.run.trial183:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.811 DEBUG:teuthology.orchestra.run.trial191:> sudo hostname $(hostname -s) 2026-02-24T16:39:46.837 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2026-02-24T16:39:46.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:46.959 INFO:tasks.cephadm:builder_project result: [{'url': 'https://1.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/flavors/default/', 'ref': 'wip-yuri8-testing-2026-02-23-2128', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'distro': 'centos', 'distro_version': '9', 'distro_codename': None, 'modified': '2026-02-24 07:28:40.641130', 'status': 'ready', 'flavor': 'default', 'project': 'ceph', 'archs': ['x86_64', 'source'], 'extra': {'version': '20.3.0-5507-g618c318a', 'package_manager_version': '20.3.0-5507.g618c318a', 'build_url': 'https://jenkins.ceph.com/job/ceph-dev-pipeline/3225/', 'root_build_cause': '', 'node_name': '10.20.192.91+toko01', 'job_name': 'ceph-dev-pipeline'}}] 2026-02-24T16:39:47.098 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-yuri8-testing-2026-02-23-2128, sha1 618c318a7a0b287ab5aeff207b7bf1d73ef45044 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:39:47.100 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm 2026-02-24T16:39:47.100 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm 2026-02-24T16:39:47.100 DEBUG:teuthology.orchestra.run.trial051:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:47.404 INFO:teuthology.orchestra.run.trial051.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:47.404 DEBUG:teuthology.orchestra.run.trial052:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:47.712 INFO:teuthology.orchestra.run.trial052.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:47.712 DEBUG:teuthology.orchestra.run.trial151:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.007 INFO:teuthology.orchestra.run.trial151.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.007 DEBUG:teuthology.orchestra.run.trial153:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.288 INFO:teuthology.orchestra.run.trial153.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.288 DEBUG:teuthology.orchestra.run.trial183:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.580 INFO:teuthology.orchestra.run.trial183.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.581 DEBUG:teuthology.orchestra.run.trial191:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.906 INFO:teuthology.orchestra.run.trial191.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076387 Feb 24 16:39 /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.906 DEBUG:teuthology.orchestra.run.trial051:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.925 DEBUG:teuthology.orchestra.run.trial052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.945 DEBUG:teuthology.orchestra.run.trial151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.964 DEBUG:teuthology.orchestra.run.trial153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:48.984 DEBUG:teuthology.orchestra.run.trial183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:49.003 DEBUG:teuthology.orchestra.run.trial191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-24T16:39:49.046 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 on all hosts... 2026-02-24T16:39:49.047 DEBUG:teuthology.orchestra.run.trial051:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.050 DEBUG:teuthology.orchestra.run.trial052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.052 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.055 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.058 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.060 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 pull 2026-02-24T16:39:49.189 INFO:teuthology.orchestra.run.trial051.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:49.193 INFO:teuthology.orchestra.run.trial052.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:49.198 INFO:teuthology.orchestra.run.trial183.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:49.199 INFO:teuthology.orchestra.run.trial153.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:49.202 INFO:teuthology.orchestra.run.trial191.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:49.213 INFO:teuthology.orchestra.run.trial151.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:39:59.415 INFO:teuthology.orchestra.run.trial153.stdout:{ 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout: "repo_digests": [ 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout: ] 2026-02-24T16:39:59.416 INFO:teuthology.orchestra.run.trial153.stdout:} 2026-02-24T16:39:59.513 INFO:teuthology.orchestra.run.trial191.stdout:{ 2026-02-24T16:39:59.513 INFO:teuthology.orchestra.run.trial191.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:39:59.513 INFO:teuthology.orchestra.run.trial191.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:39:59.514 INFO:teuthology.orchestra.run.trial191.stdout: "repo_digests": [ 2026-02-24T16:39:59.514 INFO:teuthology.orchestra.run.trial191.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:39:59.514 INFO:teuthology.orchestra.run.trial191.stdout: ] 2026-02-24T16:39:59.514 INFO:teuthology.orchestra.run.trial191.stdout:} 2026-02-24T16:39:59.658 INFO:teuthology.orchestra.run.trial151.stdout:{ 2026-02-24T16:39:59.658 INFO:teuthology.orchestra.run.trial151.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:39:59.658 INFO:teuthology.orchestra.run.trial151.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:39:59.658 INFO:teuthology.orchestra.run.trial151.stdout: "repo_digests": [ 2026-02-24T16:39:59.659 INFO:teuthology.orchestra.run.trial151.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:39:59.659 INFO:teuthology.orchestra.run.trial151.stdout: ] 2026-02-24T16:39:59.659 INFO:teuthology.orchestra.run.trial151.stdout:} 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout: "repo_digests": [ 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout: ] 2026-02-24T16:40:00.208 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-24T16:40:01.710 INFO:teuthology.orchestra.run.trial183.stdout:{ 2026-02-24T16:40:01.710 INFO:teuthology.orchestra.run.trial183.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:40:01.710 INFO:teuthology.orchestra.run.trial183.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:40:01.711 INFO:teuthology.orchestra.run.trial183.stdout: "repo_digests": [ 2026-02-24T16:40:01.711 INFO:teuthology.orchestra.run.trial183.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:40:01.711 INFO:teuthology.orchestra.run.trial183.stdout: ] 2026-02-24T16:40:01.711 INFO:teuthology.orchestra.run.trial183.stdout:} 2026-02-24T16:40:03.648 INFO:teuthology.orchestra.run.trial052.stdout:{ 2026-02-24T16:40:03.648 INFO:teuthology.orchestra.run.trial052.stdout: "ceph_version": "ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev)", 2026-02-24T16:40:03.649 INFO:teuthology.orchestra.run.trial052.stdout: "image_id": "c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7", 2026-02-24T16:40:03.649 INFO:teuthology.orchestra.run.trial052.stdout: "repo_digests": [ 2026-02-24T16:40:03.649 INFO:teuthology.orchestra.run.trial052.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18" 2026-02-24T16:40:03.649 INFO:teuthology.orchestra.run.trial052.stdout: ] 2026-02-24T16:40:03.649 INFO:teuthology.orchestra.run.trial052.stdout:} 2026-02-24T16:40:03.666 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.695 DEBUG:teuthology.orchestra.run.trial052:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.723 DEBUG:teuthology.orchestra.run.trial151:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.753 DEBUG:teuthology.orchestra.run.trial153:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.780 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.809 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /etc/ceph 2026-02-24T16:40:03.837 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.864 DEBUG:teuthology.orchestra.run.trial052:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.890 DEBUG:teuthology.orchestra.run.trial151:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.917 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.944 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.969 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 777 /etc/ceph 2026-02-24T16:40:03.994 INFO:tasks.cephadm:Writing seed config... 2026-02-24T16:40:03.996 INFO:tasks.cephadm: override: [global] mon warn on pool no app = False 2026-02-24T16:40:03.996 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2026-02-24T16:40:03.996 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [mon] debug mon = 20 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [mon] debug ms = 1 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [osd] debug ms = 1 2026-02-24T16:40:03.997 INFO:tasks.cephadm: override: [osd] debug osd = 20 2026-02-24T16:40:03.998 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:03.998 DEBUG:teuthology.orchestra.run.trial151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2026-02-24T16:40:04.016 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6deba16a-119f-11f1-a847-d404e6e7d460 mon warn on pool no app = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd mclock skip benchmark = true osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2026-02-24T16:40:04.016 DEBUG:teuthology.orchestra.run.trial151:mon.a> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service 2026-02-24T16:40:04.058 DEBUG:teuthology.orchestra.run.trial151:mgr.x> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x.service 2026-02-24T16:40:04.099 INFO:tasks.cephadm:Bootstrapping... 2026-02-24T16:40:04.100 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 -v bootstrap --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 10.20.193.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:04.239 INFO:teuthology.orchestra.run.trial151.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:40:04.239 INFO:teuthology.orchestra.run.trial151.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044', '-v', 'bootstrap', '--fsid', '6deba16a-119f-11f1-a847-d404e6e7d460', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '10.20.193.151', '--skip-admin-label'] 2026-02-24T16:40:04.240 INFO:teuthology.orchestra.run.trial151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2026-02-24T16:40:04.240 INFO:teuthology.orchestra.run.trial151.stdout:Verifying podman|docker is present... 2026-02-24T16:40:04.254 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stdout 5.8.0 2026-02-24T16:40:04.254 INFO:teuthology.orchestra.run.trial151.stdout:Verifying lvm2 is present... 2026-02-24T16:40:04.254 INFO:teuthology.orchestra.run.trial151.stdout:Verifying time synchronization is in place... 2026-02-24T16:40:04.260 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-24T16:40:04.260 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-24T16:40:04.266 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-24T16:40:04.266 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:04.272 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout enabled 2026-02-24T16:40:04.279 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout active 2026-02-24T16:40:04.279 INFO:teuthology.orchestra.run.trial151.stdout:Unit chronyd.service is enabled and running 2026-02-24T16:40:04.279 INFO:teuthology.orchestra.run.trial151.stdout:Repeating the final host check... 2026-02-24T16:40:04.293 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stdout 5.8.0 2026-02-24T16:40:04.293 INFO:teuthology.orchestra.run.trial151.stdout:podman (/bin/podman) version 5.8.0 is present 2026-02-24T16:40:04.293 INFO:teuthology.orchestra.run.trial151.stdout:systemctl is present 2026-02-24T16:40:04.293 INFO:teuthology.orchestra.run.trial151.stdout:lvcreate is present 2026-02-24T16:40:04.298 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-24T16:40:04.298 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-24T16:40:04.304 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-24T16:40:04.304 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:04.310 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout enabled 2026-02-24T16:40:04.316 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout active 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Unit chronyd.service is enabled and running 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Host looks OK 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Cluster fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Acquiring lock 139914519510032 on /run/cephadm/6deba16a-119f-11f1-a847-d404e6e7d460.lock 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Lock 139914519510032 acquired on /run/cephadm/6deba16a-119f-11f1-a847-d404e6e7d460.lock 2026-02-24T16:40:04.317 INFO:teuthology.orchestra.run.trial151.stdout:Verifying IP 10.20.193.151 port 3300 ... 2026-02-24T16:40:04.318 INFO:teuthology.orchestra.run.trial151.stdout:Verifying IP 10.20.193.151 port 6789 ... 2026-02-24T16:40:04.318 INFO:teuthology.orchestra.run.trial151.stdout:Base mon IP(s) is [10.20.193.151:3300, 10.20.193.151:6789], mon addrv is [v2:10.20.193.151:3300,v1:10.20.193.151:6789] 2026-02-24T16:40:04.320 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout default via 10.20.192.1 dev enp1s0f0 proto dhcp src 10.20.193.151 metric 102 2026-02-24T16:40:04.320 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout 10.20.192.0/20 dev enp1s0f0 proto kernel scope link src 10.20.193.151 metric 102 2026-02-24T16:40:04.321 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout 169.254.3.0/24 dev enp7s0f4u2u2c2 proto kernel scope link src 169.254.3.1 metric 101 2026-02-24T16:40:04.322 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2026-02-24T16:40:04.323 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout fe80::/64 dev enp1s0f0 proto kernel metric 256 pref medium 2026-02-24T16:40:04.323 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout fe80::/64 dev enp7s0f4u2u2c2 proto kernel metric 1024 pref medium 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout 2: enp7s0f4u2u2c2: mtu 1500 state UNKNOWN qlen 1000 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout inet6 fe80::b678:5649:6955:7250/64 scope link noprefixroute 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-24T16:40:04.326 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout 3: enp1s0f0: mtu 1500 state UP qlen 1000 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout inet6 fe80::925a:8ff:fe77:6454/64 scope link 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:Mon IP `10.20.193.151` is in CIDR network `10.20.192.0/20` 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:Mon IP `10.20.193.151` is in CIDR network `10.20.192.0/20` 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:Inferred mon public CIDR from local network configuration ['10.20.192.0/20', '10.20.192.0/20'] 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2026-02-24T16:40:04.327 INFO:teuthology.orchestra.run.trial151.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stdout c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044... 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Getting image source signatures 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Copying blob sha256:747ce4acc101875dc40b315334bcca7e1ed56a66e42d321d587adefa2486b5b8 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Copying blob sha256:8e380faede39ebd4286247457b408d979ab568aafd8389c42ec304b8cfba4e92 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Copying config sha256:c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 2026-02-24T16:40:04.606 INFO:teuthology.orchestra.run.trial151.stdout:/bin/podman: stderr Writing manifest to image destination 2026-02-24T16:40:04.747 INFO:teuthology.orchestra.run.trial151.stdout:ceph: stdout ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev) 2026-02-24T16:40:04.747 INFO:teuthology.orchestra.run.trial151.stdout:Ceph version: ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev) 2026-02-24T16:40:04.747 INFO:teuthology.orchestra.run.trial151.stdout:Extracting ceph user uid/gid from container image... 2026-02-24T16:40:04.866 INFO:teuthology.orchestra.run.trial151.stdout:stat: stdout 167 167 2026-02-24T16:40:04.866 INFO:teuthology.orchestra.run.trial151.stdout:Creating initial keys... 2026-02-24T16:40:04.996 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph-authtool: stdout AQBk1J1p/oS0OhAAIlh2c1g1J+v6nZWqNxQGkQ== 2026-02-24T16:40:05.114 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph-authtool: stdout AQBl1J1pIcgeBhAAxTNBW+4iQ0HIWLPIRFC7HQ== 2026-02-24T16:40:05.258 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph-authtool: stdout AQBl1J1p6uCaDhAAyAeObfw3FEgxma90oE0tqg== 2026-02-24T16:40:05.258 INFO:teuthology.orchestra.run.trial151.stdout:Creating initial monmap... 2026-02-24T16:40:05.400 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2026-02-24T16:40:05.400 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = tentacle 2026-02-24T16:40:05.400 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:monmaptool for a [v2:10.20.193.151:3300,v1:10.20.193.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:setting min_mon_release = tentacle 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: set fsid to 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:05.401 INFO:teuthology.orchestra.run.trial151.stdout:Creating mon... 2026-02-24T16:40:05.532 INFO:teuthology.orchestra.run.trial151.stdout:create mon.a on 2026-02-24T16:40:05.712 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2026-02-24T16:40:05.812 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2026-02-24T16:40:05.921 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target → /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target. 2026-02-24T16:40:05.922 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target → /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target. 2026-02-24T16:40:06.045 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a 2026-02-24T16:40:06.046 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service: Unit ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service not loaded. 2026-02-24T16:40:06.164 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target.wants/ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service → /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460@.service. 2026-02-24T16:40:06.310 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-24T16:40:06.310 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout disabled 2026-02-24T16:40:06.316 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-24T16:40:06.316 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:06.317 INFO:teuthology.orchestra.run.trial151.stdout:firewalld.service is not enabled 2026-02-24T16:40:06.317 INFO:teuthology.orchestra.run.trial151.stdout:Not possible to enable service . firewalld.service is not available 2026-02-24T16:40:06.317 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mon to start... 2026-02-24T16:40:06.317 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mon... 2026-02-24T16:40:06.323 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 systemd[1]: Started Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:40:06.473 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout cluster: 2026-02-24T16:40:06.473 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout id: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:06.473 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2026-02-24T16:40:06.473 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.473 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout services: 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.132118s) [leader: a] 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout data: 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout pgs: 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.474 INFO:teuthology.orchestra.run.trial151.stdout:mon is available 2026-02-24T16:40:06.475 INFO:teuthology.orchestra.run.trial151.stdout:Assimilating anything we can from ceph.conf... 2026-02-24T16:40:06.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 ceph-mon[39027]: mkfs 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:06.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 ceph-mon[39027]: mon.a is new leader, mons a in quorum (ranks 0) 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [global] 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout fsid = 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_host = [v2:10.20.193.151:3300,v1:10.20.193.151:6789] 2026-02-24T16:40:06.625 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [mgr] 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [osd] 2026-02-24T16:40:06.626 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-24T16:40:06.627 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-24T16:40:06.627 INFO:teuthology.orchestra.run.trial151.stdout:Generating new minimal ceph.conf... 2026-02-24T16:40:06.774 INFO:teuthology.orchestra.run.trial151.stdout:Restarting the monitor... 2026-02-24T16:40:06.860 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 systemd[1]: Stopping Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a[39023]: 2026-02-24T16:40:06.859+0000 7ff37efda640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a[39023]: 2026-02-24T16:40:06.859+0000 7ff37efda640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 podman[39299]: 2026-02-24 16:40:06.960771443 +0000 UTC m=+0.113700626 container died 848c364b926494e2721b2b8fc602c9cd95c477d630b19188f2dec25e7355a1ea (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, io.buildah.version=1.33.7, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.schema-version=1.0) 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 podman[39299]: 2026-02-24 16:40:06.969147511 +0000 UTC m=+0.122076704 container remove 848c364b926494e2721b2b8fc602c9cd95c477d630b19188f2dec25e7355a1ea (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.build-date=20260223, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.license=GPLv2, io.buildah.version=1.33.7, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:06 trial151 bash[39299]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service: Deactivated successfully. 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 systemd[1]: Stopped Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:40:07.122 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 systemd[1]: Starting Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:40:07.123 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 podman[39423]: 2026-02-24 16:40:07.122064722 +0000 UTC m=+0.014917760 container create 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.build-date=20260223, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, io.buildah.version=1.33.7, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-24T16:40:07.157 INFO:teuthology.orchestra.run.trial151.stdout:Setting public_network to 10.20.192.0/20 in global config section 2026-02-24T16:40:07.308 INFO:teuthology.orchestra.run.trial151.stdout:Wrote config to /etc/ceph/ceph.conf 2026-02-24T16:40:07.309 INFO:teuthology.orchestra.run.trial151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:07.309 INFO:teuthology.orchestra.run.trial151.stdout:Creating mgr... 2026-02-24T16:40:07.309 INFO:teuthology.orchestra.run.trial151.stdout:Verifying port 0.0.0.0:9283 ... 2026-02-24T16:40:07.309 INFO:teuthology.orchestra.run.trial151.stdout:Verifying port 0.0.0.0:8765 ... 2026-02-24T16:40:07.445 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x 2026-02-24T16:40:07.445 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x.service: Unit ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x.service not loaded. 2026-02-24T16:40:07.451 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 podman[39423]: 2026-02-24 16:40:07.149753207 +0000 UTC m=+0.042606245 container init 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260223, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, ceph=True, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-yuri8-testing-2026-02-23-2128) 2026-02-24T16:40:07.451 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 podman[39423]: 2026-02-24 16:40:07.152623375 +0000 UTC m=+0.045476423 container start 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.33.7, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260223, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.license=GPLv2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, CEPH_REF=wip-yuri8-testing-2026-02-23-2128) 2026-02-24T16:40:07.451 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 bash[39423]: 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 podman[39423]: 2026-02-24 16:40:07.11600072 +0000 UTC m=+0.008853768 image pull c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 systemd[1]: Started Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: set uid:gid to 167:167 (ceph:ceph) 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: ceph version 20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev - RelWithDebInfo), process ceph-mon, pid 2 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: pidfile_write: ignore empty --pid-file 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: load: jerasure load: lrc 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: RocksDB version: 7.9.2 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Git sha 0 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Compile date 2026-02-24 06:58:23 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: DB SUMMARY 2026-02-24T16:40:07.452 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: DB Session ID: SA89U59L51ECNM31GDCQ 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: CURRENT file: CURRENT 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: IDENTITY file: IDENTITY 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 77693 ; 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.error_if_exists: 0 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.create_if_missing: 0 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.paranoid_checks: 1 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.flush_verify_memtable_count: 1 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2026-02-24T16:40:07.453 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.env: 0xee45a0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.fs: PosixFileSystem 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.info_log: 0x349ed60 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_file_opening_threads: 16 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.statistics: (nil) 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.use_fsync: 0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_log_file_size: 0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_manifest_file_size: 1073741824 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.log_file_time_to_roll: 0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.keep_log_file_num: 1000 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.recycle_log_file_num: 0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_fallocate: 1 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_mmap_reads: 0 2026-02-24T16:40:07.454 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_mmap_writes: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.use_direct_reads: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.create_missing_column_families: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.db_log_dir: 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.wal_dir: 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.table_cache_numshardbits: 6 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.WAL_ttl_seconds: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.WAL_size_limit_MB: 0 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.manifest_preallocation_size: 4194304 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.is_fd_close_on_exec: 1 2026-02-24T16:40:07.455 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.advise_random_on_open: 1 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.db_write_buffer_size: 0 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.write_buffer_manager: 0x349a140 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.access_hint_on_compaction_start: 1 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.use_adaptive_mutex: 0 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.rate_limiter: (nil) 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.wal_recovery_mode: 2 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enable_thread_tracking: 0 2026-02-24T16:40:07.456 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enable_pipelined_write: 0 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.unordered_write: 0 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.write_thread_max_yield_usec: 100 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.row_cache: None 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.wal_filter: None 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.avoid_flush_during_recovery: 0 2026-02-24T16:40:07.458 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_ingest_behind: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.two_write_queues: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.manual_wal_flush: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.wal_compression: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.atomic_flush: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.persist_stats_to_disk: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.write_dbid_to_manifest: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.log_readahead_size: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.best_efforts_recovery: 0 2026-02-24T16:40:07.459 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.allow_data_in_errors: 0 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.db_host_id: __hostname__ 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enforce_single_del_contracts: true 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_background_jobs: 2 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_background_compactions: -1 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_subcompactions: 1 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.delayed_write_rate : 16777216 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_total_wal_size: 0 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2026-02-24T16:40:07.460 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.stats_dump_period_sec: 600 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.stats_persist_period_sec: 600 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.stats_history_buffer_size: 1048576 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_open_files: -1 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bytes_per_sync: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.wal_bytes_per_sync: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.strict_bytes_per_sync: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_readahead_size: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_background_flushes: -1 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Compression algorithms supported: 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kZSTD supported: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kXpressCompression supported: 0 2026-02-24T16:40:07.461 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kBZip2Compression supported: 0 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kZSTDNotFinalCompression supported: 0 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kLZ4Compression supported: 1 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kZlibCompression supported: 1 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kLZ4HCCompression supported: 1 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: kSnappyCompression supported: 1 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Fast CRC32 supported: Supported on x86 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: DMutex implementation: pthread_mutex_t 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.merge_operator: 2026-02-24T16:40:07.462 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_filter: None 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_filter_factory: None 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.sst_partitioner_factory: None 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.memtable_factory: SkipListFactory 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.table_factory: BlockBasedTable 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x349f500) 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: cache_index_and_filter_blocks: 1 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: pin_top_level_index_and_filter: 1 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: index_type: 0 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: data_block_index_type: 0 2026-02-24T16:40:07.463 INFO:journalctl@ceph.mon.a.trial151.stdout: index_shortening: 1 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: data_block_hash_table_util_ratio: 0.750000 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: checksum: 4 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: no_block_cache: 0 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: block_cache: 0x346da50 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: block_cache_name: BinnedLRUCache 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: block_cache_options: 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: capacity : 536870912 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: num_shard_bits : 4 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: strict_capacity_limit : 0 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: high_pri_pool_ratio: 0.000 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: block_cache_compressed: (nil) 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: persistent_cache: (nil) 2026-02-24T16:40:07.464 INFO:journalctl@ceph.mon.a.trial151.stdout: block_size: 4096 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: block_size_deviation: 10 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: block_restart_interval: 16 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: index_block_restart_interval: 1 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: metadata_block_size: 4096 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: partition_filters: 0 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: use_delta_encoding: 1 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: filter_policy: bloomfilter 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: whole_key_filtering: 1 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: verify_compression: 0 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: read_amp_bytes_per_bit: 0 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: format_version: 5 2026-02-24T16:40:07.465 INFO:journalctl@ceph.mon.a.trial151.stdout: enable_index_compression: 1 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout: block_align: 0 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout: max_auto_readahead_size: 262144 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout: prepopulate_block_cache: 0 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout: initial_auto_readahead_size: 8192 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout: num_file_reads_for_auto_readahead: 2 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.write_buffer_size: 33554432 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_write_buffer_number: 2 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression: NoCompression 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression: Disabled 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.prefix_extractor: nullptr 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2026-02-24T16:40:07.466 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.num_levels: 7 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2026-02-24T16:40:07.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.window_bits: -14 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.level: 32767 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.strategy: 0 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.parallel_threads: 1 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.enabled: false 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2026-02-24T16:40:07.468 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.level0_stop_writes_trigger: 36 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.target_file_size_base: 67108864 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.target_file_size_multiplier: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2026-02-24T16:40:07.469 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_compaction_bytes: 1677721600 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.arena_block_size: 1048576 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.disable_auto_compactions: 0 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2026-02-24T16:40:07.470 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.inplace_update_support: 0 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.inplace_update_num_locks: 10000 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.memtable_whole_key_filtering: 0 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.memtable_huge_page_size: 0 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.bloom_locality: 0 2026-02-24T16:40:07.471 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.max_successive_merges: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.optimize_filters_for_hits: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.paranoid_file_checks: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.force_consistency_checks: 1 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.report_bg_io_stats: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.ttl: 2592000 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.periodic_compaction_seconds: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.preclude_last_level_data_seconds: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.preserve_internal_time_seconds: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enable_blob_files: false 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.min_blob_size: 0 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_file_size: 268435456 2026-02-24T16:40:07.472 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_compression_type: NoCompression 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.enable_blob_garbage_collection: false 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_compaction_readahead_size: 0 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.blob_file_starting_level: 0 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8a4e218b-adab-44ac-b4ee-c6d074871cfe 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771951207169188, "job": 1, "event": "recovery_started", "wal_files": [9]} 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771951207170097, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 74669, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 231, "table_properties": {"data_size": 72944, "index_size": 174, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 9888, "raw_average_key_size": 48, "raw_value_size": 67379, "raw_average_value_size": 333, "num_data_blocks": 8, "num_entries": 202, "num_filter_entries": 202, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1771951207, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8a4e218b-adab-44ac-b4ee-c6d074871cfe", "db_session_id": "SA89U59L51ECNM31GDCQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2026-02-24T16:40:07.473 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771951207170159, "job": 1, "event": "recovery_finished"} 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x34bae00 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: DB pointer 0x42ac000 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: rocksdb: [db/db_impl/db_impl.cc:1111] 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout: ** DB Stats ** 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-24T16:40:07.474 INFO:journalctl@ceph.mon.a.trial151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: ** Compaction Stats [default] ** 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: L0 2/0 74.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Sum 2/0 74.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.475 INFO:journalctl@ceph.mon.a.trial151.stdout: ** Compaction Stats [default] ** 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 111.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: AddFile(Total Files): cumulative 0, interval 0 2026-02-24T16:40:07.476 INFO:journalctl@ceph.mon.a.trial151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: AddFile(Keys): cumulative 0, interval 0 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: Cumulative compaction: 0.00 GB write, 25.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: Interval compaction: 0.00 GB write, 25.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: Block cache BinnedLRUCache@0x346da50#2 capacity: 512.00 MB usage: 1.06 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.70 KB,0.00013411%) IndexBlock(2,0.36 KB,6.85453e-05%) Misc(1,0.00 KB,0%) 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout: ** File Read Latency Histogram By Level [default] ** 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: starting mon.a rank 0 at public addrs [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] at bind addrs [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???) e1 preinit fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).mds e1 new map 2026-02-24T16:40:07.477 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).mds e1 print_map 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: e1 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: btime 2026-02-24T16:40:06:321537+0000 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: legacy client fscid: -1 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout: No filesystems configured 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-24T16:40:07.478 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mon.a is new leader, mons a in quorum (ranks 0) 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: monmap epoch 1 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: last_changed 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: min_mon_release 20 (tentacle) 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: election_strategy: 1 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: fsmap 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: osdmap e1: 0 total, 0 up, 0 in 2026-02-24T16:40:07.479 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:07 trial151 ceph-mon[39453]: mgrmap e1: no daemons active 2026-02-24T16:40:07.558 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460.target.wants/ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x.service → /etc/systemd/system/ceph-6deba16a-119f-11f1-a847-d404e6e7d460@.service. 2026-02-24T16:40:07.698 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-24T16:40:07.699 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout disabled 2026-02-24T16:40:07.704 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-24T16:40:07.704 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:07.704 INFO:teuthology.orchestra.run.trial151.stdout:firewalld.service is not enabled 2026-02-24T16:40:07.704 INFO:teuthology.orchestra.run.trial151.stdout:Not possible to enable service . firewalld.service is not available 2026-02-24T16:40:07.711 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-24T16:40:07.712 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout disabled 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:firewalld.service is not enabled 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mgr to start... 2026-02-24T16:40:07.718 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mgr... 2026-02-24T16:40:07.737 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:07 trial151 systemd[1]: Started Ceph mgr.x for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:40:07.878 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsid": "6deba16a-119f-11f1-a847-d404e6e7d460", 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "health": { 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 0 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:07.880 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "a" 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_age": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "btime": "2026-02-24T16:40:06:321537+0000", 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "available": false, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-24T16:40:07.881 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modified": "2026-02-24T16:40:06.321963+0000", 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:07.882 INFO:teuthology.orchestra.run.trial151.stdout:mgr not available, waiting (1/15)... 2026-02-24T16:40:08.692 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2160734122' entity='client.admin' 2026-02-24T16:40:08.692 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/770035416' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-24T16:40:08.987 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:08 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:40:08.988 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:08 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:40:08.988 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:08 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: from numpy import show_config as show_numpy_config 2026-02-24T16:40:10.043 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsid": "6deba16a-119f-11f1-a847-d404e6e7d460", 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "health": { 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-24T16:40:10.044 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 0 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "a" 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-24T16:40:10.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-24T16:40:10.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "btime": "2026-02-24T16:40:06:321537+0000", 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-24T16:40:10.047 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "available": false, 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modified": "2026-02-24T16:40:06.321963+0000", 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:10.048 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-24T16:40:10.049 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:10.049 INFO:teuthology.orchestra.run.trial151.stdout:mgr not available, waiting (2/15)... 2026-02-24T16:40:10.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:10 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2982022192' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-24T16:40:11.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: Activating manager daemon x 2026-02-24T16:40:11.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: mgrmap e2: x(active, starting, since 0.00253281s) 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: Manager daemon x is now available 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' 2026-02-24T16:40:11.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' 2026-02-24T16:40:11.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:11 trial151 ceph-mon[39453]: from='mgr.14100 10.20.193.151:0/131247440' entity='mgr.x' 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsid": "6deba16a-119f-11f1-a847-d404e6e7d460", 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "health": { 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-24T16:40:12.310 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 0 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "a" 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:12.311 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-24T16:40:12.312 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:12.313 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "btime": "2026-02-24T16:40:06:321537+0000", 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ], 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:12.314 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "modified": "2026-02-24T16:40:06.321963+0000", 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout }, 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:12.315 INFO:teuthology.orchestra.run.trial151.stdout:mgr is available 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [global] 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout fsid = 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_host = [v2:10.20.193.151:3300,v1:10.20.193.151:6789] 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [mgr] 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout [osd] 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-24T16:40:12.529 INFO:teuthology.orchestra.run.trial151.stdout:Enabling cephadm module... 2026-02-24T16:40:12.581 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:12 trial151 ceph-mon[39453]: mgrmap e3: x(active, since 1.00455s) 2026-02-24T16:40:12.581 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1421667507' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-24T16:40:13.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:13 trial151 ceph-mon[39453]: mgrmap e4: x(active, since 2s) 2026-02-24T16:40:13.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3141252400' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2026-02-24T16:40:13.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1943021984' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2026-02-24T16:40:13.738 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: ignoring --setuser ceph since I am not root 2026-02-24T16:40:13.738 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: ignoring --setgroup ceph since I am not root 2026-02-24T16:40:13.768 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:13.768 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "active_name": "x", 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for the mgr to restart... 2026-02-24T16:40:13.769 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mgr epoch 5... 2026-02-24T16:40:14.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:14 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1943021984' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2026-02-24T16:40:14.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:14 trial151 ceph-mon[39453]: mgrmap e5: x(active, since 3s) 2026-02-24T16:40:14.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:14 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2903890787' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2026-02-24T16:40:14.738 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:40:14.738 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:40:14.738 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: from numpy import show_config as show_numpy_config 2026-02-24T16:40:16.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: Active manager daemon x restarted 2026-02-24T16:40:16.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: Activating manager daemon x 2026-02-24T16:40:16.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: osdmap e2: 0 total, 0 up, 0 in 2026-02-24T16:40:16.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: mgrmap e6: x(active, starting, since 0.00250933s) 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-24T16:40:16.738 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:16 trial151 ceph-mon[39453]: Manager daemon x is now available 2026-02-24T16:40:17.293 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:17.294 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2026-02-24T16:40:17.294 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "initialized": true 2026-02-24T16:40:17.294 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:17.294 INFO:teuthology.orchestra.run.trial151.stdout:mgr epoch 5 is available 2026-02-24T16:40:17.294 INFO:teuthology.orchestra.run.trial151.stdout:Verifying orchestrator module is enabled... 2026-02-24T16:40:18.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: Found migration_current of "None". Setting to last migration. 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:18.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: mgrmap e7: x(active, since 1.00379s) 2026-02-24T16:40:18.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2026-02-24T16:40:18.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2026-02-24T16:40:18.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:17] ENGINE Bus STARTING 2026-02-24T16:40:18.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/867344074' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2026-02-24T16:40:18.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:17 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:18.289 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2026-02-24T16:40:18.289 INFO:teuthology.orchestra.run.trial151.stdout:Setting orchestrator backend to cephadm... 2026-02-24T16:40:18.743 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout value unchanged 2026-02-24T16:40:18.743 INFO:teuthology.orchestra.run.trial151.stdout:Generating ssh key... 2026-02-24T16:40:19.029 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Generating public/private ed25519 key pair. 2026-02-24T16:40:19.029 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Your identification has been saved in /tmp/tmpth8dss75/key 2026-02-24T16:40:19.029 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Your public key has been saved in /tmp/tmpth8dss75/key.pub 2026-02-24T16:40:19.029 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: The key fingerprint is: 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: SHA256:ZOFz/a25rDn+L+HlZhZo0b+lSthOv49kQ6mOoC3rmKI ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: The key's randomart image is: 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: +--[ED25519 256]--+ 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | . | 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | . . . | 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | = . . . | 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | o o o + | 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | S * o| 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | o =.++| 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | . . *.*=+| 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: | . o.o . *.*+==| 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: |E. .o.+o. ..B=BOo| 2026-02-24T16:40:19.030 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: +----[SHA256]-----+ 2026-02-24T16:40:19.198 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:19.199 INFO:teuthology.orchestra.run.trial151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:40:19.199 INFO:teuthology.orchestra.run.trial151.stdout:Adding key to root@localhost authorized_keys... 2026-02-24T16:40:19.199 INFO:teuthology.orchestra.run.trial151.stdout:Adding host trial151... 2026-02-24T16:40:19.312 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:17] ENGINE Serving on http://10.20.193.151:8765 2026-02-24T16:40:19.312 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:17] ENGINE Serving on https://10.20.193.151:7150 2026-02-24T16:40:19.312 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:17] ENGINE Bus STARTED 2026-02-24T16:40:19.312 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:17] ENGINE Client ('10.20.193.151', 58976) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)') 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/867344074' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: mgrmap e8: x(active, since 2s) 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:19.313 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:19 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:20.610 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:20 trial151 ceph-mon[39453]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:20.611 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:20 trial151 ceph-mon[39453]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:20.611 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:20 trial151 ceph-mon[39453]: Generating ssh key... 2026-02-24T16:40:20.611 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:20 trial151 ceph-mon[39453]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:20.611 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:20 trial151 ceph-mon[39453]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial151", "addr": "10.20.193.151", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:20.963 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout Added host 'trial151' with addr '10.20.193.151' 2026-02-24T16:40:20.963 INFO:teuthology.orchestra.run.trial151.stdout:Deploying unmanaged mon service... 2026-02-24T16:40:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:21 trial151 ceph-mon[39453]: Deploying cephadm binary to trial151 2026-02-24T16:40:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:21 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:21 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:21 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:21.684 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2026-02-24T16:40:21.684 INFO:teuthology.orchestra.run.trial151.stdout:Deploying unmanaged mgr service... 2026-02-24T16:40:22.126 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2026-02-24T16:40:22.349 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:22 trial151 ceph-mon[39453]: Added host trial151 2026-02-24T16:40:22.349 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:22 trial151 ceph-mon[39453]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:22.349 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:22 trial151 ceph-mon[39453]: Saving service mon spec with placement count:5 2026-02-24T16:40:22.349 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:22 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:22.350 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:22 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:22.627 INFO:teuthology.orchestra.run.trial151.stdout:Enabling the dashboard module... 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: Saving service mgr spec with placement count:2 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3294296118' entity='client.admin' 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/435579603' entity='client.admin' 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/622921351' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2026-02-24T16:40:23.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:23 trial151 ceph-mon[39453]: from='mgr.14118 10.20.193.151:0/1647431125' entity='mgr.x' 2026-02-24T16:40:24.023 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: ignoring --setuser ceph since I am not root 2026-02-24T16:40:24.023 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: ignoring --setgroup ceph since I am not root 2026-02-24T16:40:24.045 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "active_name": "x", 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for the mgr to restart... 2026-02-24T16:40:24.046 INFO:teuthology.orchestra.run.trial151.stdout:Waiting for mgr epoch 9... 2026-02-24T16:40:25.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:24 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/622921351' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2026-02-24T16:40:25.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:24 trial151 ceph-mon[39453]: mgrmap e9: x(active, since 7s) 2026-02-24T16:40:25.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:24 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3920470305' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2026-02-24T16:40:25.238 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:40:25.238 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:40:25.238 INFO:journalctl@ceph.mgr.x.trial151.stdout:Feb 24 16:40:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mgr-x[39722]: from numpy import show_config as show_numpy_config 2026-02-24T16:40:26.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: Active manager daemon x restarted 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: Activating manager daemon x 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: osdmap e3: 0 total, 0 up, 0 in 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: mgrmap e10: x(active, starting, since 0.00245757s) 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-24T16:40:26.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:26 trial151 ceph-mon[39453]: Manager daemon x is now available 2026-02-24T16:40:27.591 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout { 2026-02-24T16:40:27.591 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2026-02-24T16:40:27.591 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout "initialized": true 2026-02-24T16:40:27.591 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout } 2026-02-24T16:40:27.592 INFO:teuthology.orchestra.run.trial151.stdout:mgr epoch 9 is available 2026-02-24T16:40:27.592 INFO:teuthology.orchestra.run.trial151.stdout:Using certmgr to generate dashboard self-signed certificate... 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:27.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:27 trial151 ceph-mon[39453]: mgrmap e11: x(active, since 1.0042s) 2026-02-24T16:40:28.221 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFIDCCAwigAwIBAgIUXk8r3xI56l1vmJodIvpxaQduBiswDQYJKoZIhvcNAQEL\nBQAwPDE6MDgGA1UEAwwxY2VwaGFkbS1yb290LTZkZWJhMTZhLTExOWYtMTFmMS1h\nODQ3LWQ0MDRlNmU3ZDQ2MDAeFw0yNjAyMjQxNjQwMjhaFw0yOTAyMjMxNjQwMjha\nMBgxFjAUBgNVBAMMDTEwLjIwLjE5My4xNTEwggIiMA0GCSqGSIb3DQEBAQUAA4IC\nDwAwggIKAoICAQDM2giJnk7Fz149DJ7IFEZMoE4sJuTx8YoUpOGJIjX+1SqozxQB\nGGdfgBJLacFsoUGnMrTr/D1WFjTlunSFlxXnwtlzM7ztw5IK31gKQwDdwR1f2/5+\n/L38R0swX3JerLSbeo6i+6hcf2u2/YpVaCMOLL5HhgYKZ9TSAZPVRJWC1/3nadIC\nB3Bz0lIYMfhwiuKr79+gQZ7wuZeB/xZcW34yd+nTXoARk1vqKoSOj+IIsyZnjnFp\nnki6XbWtVeGLLCEeT552sFPUby9kSjLo9uUFuZIepQEsm3iqY/Cc0KjFDjvrsc9X\niQJa3tFsoM/4zi6qfpVMOzJ0GQ2U96NcQXp11RhUWZ/5ddotX1BnQ6BM5Xiv+ERt\n7d3NxZukCRQGfRMDDfEqtLt3u9NZRZvRBC71d2nn/FAI5Bxq6KaF6As5WF5crCtL\nBgALNUJvuBD1ilZ4p5zjFlwPd55w//MUvoFHD4CRg080PD8oTj1xjVDYDp6NY6Mu\ndHw4rqVA8Accwfh9RLLZrsqjH4WVAmnhLQJcw2NZSs0tFdmtDVTcywqc/PUzewE4\nIUA2/VxUQLGJTj/PF2xQmNdbpKVx6wpQ8j4EnP5++xR7jIXC2O1Lb6FuWfIlYn+S\nswmxgemSkIXhPFZF+JKd3tTn+i089OWpHXumNxCpcpn+reaDCaeiPfQulwIDAQAB\noz4wPDAsBgNVHREEJTAjggh0cmlhbDE1MYIRZGFzaGJvYXJkX3NlcnZlcnOHBAoU\nwZcwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEAgnthl9D1g15Z03nV\ndrmJW+K2aLu9Pvsv6WT9CYiiwtaGaq20GlDcztfEADIxQonLwlq0T8ywYM7thANu\ndzDBp4bdl/BB3FwCHCCdnxcV1KO6pYszN4SF+r0XvLtlrw8R3JJN5nS+usuHiDEY\nbK95WSSfnjHXpikJQqmyFHWYlGzbD6l1FrsWF6JzD77L44XeoRLFKN9lGDV5n64B\nTPuEBtlwzfDV7cDEXrpkGJPWPuONMc0NyFFmy8PyU8Dg9Sg/AF2rr5QdKMQB1SnQ\nzLJyFJj/cjdCWxzGBs6+j1Spu7wkS4fp3AOhW2bP5lMo4BVRiWx6szqC5NeRNLVW\nDCccUPitAPeG9hIPSigLkhdlxeZcBIlDEp0TpkFu8YWieJ6g1ccWglts5nWtRP1/\nfX+FNDggABd4jO70h/38PIV578xc36vAgsWVZjvFd8dcevRMeQp2dk0K+PFpC5oQ\nY84bOM5uRGQN4rvSGJ93cNuMhZfmKZXKlQuNl0Jj5Q5zT+8r2NSm/ui5aVu9mCHJ\nSBNK1AJUQlFSIUPhmEhRJT4CPkzrWixnmD1MPampDEu/PH9pUNba+rwxTlfLGKTw\nSkZL+zi3MQl/+MR/KGJQ0ZwUwkxT5BSTUlfOo3lVNRuPZoFBxocAV8865AOKE6l8\ntoUif3HbBKTbphekTozns/N49tA=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKgIBAAKCAgEAzNoIiZ5Oxc9ePQyeyBRGTKBOLCbk8fGKFKThiSI1/tUqqM8U\nARhnX4ASS2nBbKFBpzK06/w9VhY05bp0hZcV58LZczO87cOSCt9YCkMA3cEdX9v+\nfvy9/EdLMF9yXqy0m3qOovuoXH9rtv2KVWgjDiy+R4YGCmfU0gGT1USVgtf952nS\nAgdwc9JSGDH4cIriq+/foEGe8LmXgf8WXFt+Mnfp016AEZNb6iqEjo/iCLMmZ45x\naZ5Iul21rVXhiywhHk+edrBT1G8vZEoy6PblBbmSHqUBLJt4qmPwnNCoxQ4767HP\nV4kCWt7RbKDP+M4uqn6VTDsydBkNlPejXEF6ddUYVFmf+XXaLV9QZ0OgTOV4r/hE\nbe3dzcWbpAkUBn0TAw3xKrS7d7vTWUWb0QQu9Xdp5/xQCOQcauimhegLOVheXKwr\nSwYACzVCb7gQ9YpWeKec4xZcD3eecP/zFL6BRw+AkYNPNDw/KE49cY1Q2A6ejWOj\nLnR8OK6lQPAHHMH4fUSy2a7Kox+FlQJp4S0CXMNjWUrNLRXZrQ1U3MsKnPz1M3sB\nOCFANv1cVECxiU4/zxdsUJjXW6SlcesKUPI+BJz+fvsUe4yFwtjtS2+hblnyJWJ/\nkrMJsYHpkpCF4TxWRfiSnd7U5/otPPTlqR17pjcQqXKZ/q3mgwmnoj30LpcCAwEA\nAQKCAgBKCbGVP5y73Ld0PCf1K3K/Bh9K0OHlWgWscW08WZuHV3b8HR/fR3kxKII0\nNGxevSeT7P0732Yxw4Pp1cqAb/tQUPmzq+roSktx14gj6VSaUU4zE7l3HxTxZbkU\nqiKVejkqipCb8nXGvJzrPdauCJ3G5T/+bL+pKh7eqghKx+d7q2IOrg3jgtnPWqAm\nM4VV87R6LQ+3hZAS9BkAuHfV+JFzXyipJtoFwCszxpszRWiN+WXzfqWZ5UXm6bCc\njdIfMVtk76TXw+IAgRDxX+nnjjh3KbtYjQ2QGD7z4nwXyUh+NCdPB+xm+FNcEXVC\nnBqek7KFF4wRScHx+Q4SHbCyys2dvDc4FjXHBoaqimgk6cMCWHIoM5uj35qPGei1\nAI56ehGzXywhGuT1+ZRcEohs1OsbH+s34mACIhugzf0bQbShN1aedFIvun2YoUM7\nDpLz3T0yJN8uNriAGRSVaaIpsJaHlVG7RVxrFw2/WDHOJt9YfhWWEdBHRlcPiluz\n4sy6GXaxF31pUWi3GV2g5EpUoxlLLOpmXCizAhlMTUzRbZPg+9LXupUiUXxTOtkw\nVejHMWzmFw6pYFGiCuW0tCwqpT1StqIwbPMYzrOLSOYSnJmkCeQnVsnoEjTLQ+aW\nrbhb7NWtWN+NSWg8H4oqXs44tJLZdnCoSUVvP4lpUEizSbiZWQKCAQEA7fXKqdqK\nNGJj3L/i7r9oNeuh/McEZvo3wlN6XDVze6TjPgk+NMsLpWtdkmR3UkqKjkWqLpti\n7/LbIVKmjIv7U1DgUSjZkiwy9r83QPH3k5ZY81ovaOuf0QXlpZHRkzzL7b0Q96+0\np52mzti8BIj790Qmak6yrk0/gfkPaaXm4/SWqcBURPdY5NmXjLPItbJ4dpoRf7TF\n5Y3HZPnCIuvkRY/c7VeooVSG03OQSQRGwAPOrHKvjhtiFWR5DOmKr9RsUgUURQLG\n66+f8r1vWVYmxvTTyRim/dunoN7HhtgznaUuj4/aHF4dogl3sgUBx181vXVPO8/V\nhXxqyFla+sesyQKCAQEA3GGwsWCuazz4WxefUHmiTfCeyzfm2h72ShjR6wqxBhup\nOta4W2hts+6rPgK08nxrLj5DcFZ9f62QbmVwIAcx6lDb08ZeAzu5FjIHUZn9y1Ww\ngGKCKDKiKQzMiyps36+VuenBab/MUm3fWoPyCOF1pls0rOHjED2Y/xJbfIFrO/x9\nzhcMX3RyFQ18H4saTd1f03C79JYD9AXxxEr4kIMp4LMs9c7nGJGwWztRuNxBvMcB\nrW4JIerAsDnQ9gaLuEGS7sHldFMmUtUcF0GQFwEwuHw6NM4sy/g6BUABMq7YCH55\n8dxeOy6X+3APBmxCKpZZ+E+x7mbaSbYwYaoQLlqQXwKCAQEA5jvHPzoMOAzuVBpw\nhdE6EdmPXFRuhPgZ1PIR1H9+4aWynEYzIr9F4GIWIdsEu+EVV/w9YWTEgCa4L/CY\n5ilWHNGGh9//SndrQ+PUoQi8Itb371RVxK0dbw7Gl8CTRmMvaDFi5lLhdr+JMl7u\n43BqYV5V0MueBBkHlkiUygJTeEEhONK2dhffsWR6Rg945ZvtA8ujnK0aQDVCNAeW\n91THAyrjddw3cwimfpY0hwEVNxk38Q/xnYwPBWW13XFk1ySvPVheuTqrnkPJfQIe\nZxB86kjdVJd+kuXgtTx3TpeGsUOt3gcv3/yLrUh9wAcqJm2PQGTtKZC4JbQwWPJj\nnXpuSQKCAQEAuPLjLnlM54KLiycP9KGfrHk7SqDnhDZny1d5FCAs4EtmgpKgtVW2\ne0LZQ3lSEg6IhI6MWkK7n+MxSvSmXLqtEVA6rEfVrjD467+glML/Ux3eul/xi3qy\nzHCJWGGKbU8WAKhEQVNNTgAXagdBoGJWMf5o/o2z2XVYaPgm/x49uGH10ndVUuaO\nt0xR1scOlDosqW36zxD20FwMwgeJAv6QrpqwKLozB1N9BZOKyfFO2fbJWZr23xf8\nDej3jQGBUGj3vpfK0QzXJLzkjj/fhO7izHC1CIve3G69gppsZvSFKNu7y1Z9Xn2D\n3uf0NBe2j7cOWm3HjaOTnjjf8ZafbcdjUQKCAQEAjO2FHDKQ1ycAkmgS5j1jyQFG\nfBqGmla8/lJBSlvTqTUMbX+BnjCCo2eYjQk7oNRpG6VXucXvp92qM1HSMCK8FVFc\nxwxrRj8FCU1sw8iB4yVvNL0g1urwdSc74rUUkp7aax5m+wBn4cRTDOciEVVwppTm\nYvRi11/lWGoZUAdFfQ2PzO4mIzEvOH+/W0uQyt8J8KXjVDxWUqhFMSRp1022EYFV\ngh2Uow5HmfVK03rXsxXtC37mJKyOcZaClFFlA2bhWw5laFGpLl+utxhLQWwmSNgW\nQce7qG0hIYg63o8G3g+cHN26fsjiwuHRu8s7iYeiTPbupasrYV2kzNhXcDx5oQ==\n-----END RSA PRIVATE KEY-----\n"} 2026-02-24T16:40:28.459 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout SSL certificate updated 2026-02-24T16:40:28.694 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2026-02-24T16:40:28.695 INFO:teuthology.orchestra.run.trial151.stdout:Creating initial admin user... 2026-02-24T16:40:28.716 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:26] ENGINE Bus STARTING 2026-02-24T16:40:28.716 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:27] ENGINE Serving on http://10.20.193.151:8765 2026-02-24T16:40:28.716 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:27] ENGINE Serving on https://10.20.193.151:7150 2026-02-24T16:40:28.716 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:27] ENGINE Bus STARTED 2026-02-24T16:40:28.716 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: [24/Feb/2026:16:40:27] ENGINE Client ('10.20.193.151', 52094) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)') 2026-02-24T16:40:28.717 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2026-02-24T16:40:28.717 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2026-02-24T16:40:28.717 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:29.135 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Q1wzMZC2FCluoltN0m2Ef./2NVIrIpmaHbsP/h0sS3eMYbaxOVeoi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1771951229, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2026-02-24T16:40:29.135 INFO:teuthology.orchestra.run.trial151.stdout:Fetching dashboard port number... 2026-02-24T16:40:29.362 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stdout 8443 2026-02-24T16:40:29.371 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-24T16:40:29.371 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout disabled 2026-02-24T16:40:29.378 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-24T16:40:29.378 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:29.378 INFO:teuthology.orchestra.run.trial151.stdout:firewalld.service is not enabled 2026-02-24T16:40:29.378 INFO:teuthology.orchestra.run.trial151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout:Ceph Dashboard is now available at: 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout: URL: https://trial151:8443/ 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout: User: admin 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout: Password: 7lm21wlq28 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.379 INFO:teuthology.orchestra.run.trial151.stdout:Saving cluster configuration to /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config directory 2026-02-24T16:40:29.661 INFO:teuthology.orchestra.run.trial151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2026-02-24T16:40:29.661 INFO:teuthology.orchestra.run.trial151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout:Or, if you are only running a single cluster on this host: 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2026-02-24T16:40:29.662 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout:Please consider enabling telemetry to help improve Ceph: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: ceph telemetry on 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout:For more information see: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:29.663 INFO:teuthology.orchestra.run.trial151.stdout:Bootstrap complete. 2026-02-24T16:40:29.668 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout static 2026-02-24T16:40:29.674 INFO:teuthology.orchestra.run.trial151.stdout:Non-zero exit code 3 from systemctl is-active logrotate 2026-02-24T16:40:29.675 INFO:teuthology.orchestra.run.trial151.stdout:systemctl: stdout inactive 2026-02-24T16:40:29.675 INFO:teuthology.orchestra.run.trial151.stdout:Enabling the logrotate.timer service to perform daily log rotation. 2026-02-24T16:40:29.828 INFO:tasks.cephadm:Fetching config... 2026-02-24T16:40:29.828 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:29.828 DEBUG:teuthology.orchestra.run.trial151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2026-02-24T16:40:29.845 INFO:tasks.cephadm:Fetching client.admin keyring... 2026-02-24T16:40:29.846 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:29.846 DEBUG:teuthology.orchestra.run.trial151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2026-02-24T16:40:29.909 INFO:tasks.cephadm:Fetching mon keyring... 2026-02-24T16:40:29.909 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:29.909 DEBUG:teuthology.orchestra.run.trial151:> sudo dd if=/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/keyring of=/dev/stdout 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1761069761' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: mgrmap e12: x(active, since 2s) 2026-02-24T16:40:29.912 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3462905077' entity='client.admin' 2026-02-24T16:40:29.946 INFO:tasks.cephadm:Fetching pub ssh key... 2026-02-24T16:40:29.946 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:29.946 DEBUG:teuthology.orchestra.run.trial151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2026-02-24T16:40:30.009 INFO:tasks.cephadm:Installing pub ssh key for root users... 2026-02-24T16:40:30.009 DEBUG:teuthology.orchestra.run.trial051:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.049 INFO:teuthology.orchestra.run.trial051.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.059 DEBUG:teuthology.orchestra.run.trial052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.097 INFO:teuthology.orchestra.run.trial052.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.107 DEBUG:teuthology.orchestra.run.trial151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.141 INFO:teuthology.orchestra.run.trial151.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.152 DEBUG:teuthology.orchestra.run.trial153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.187 INFO:teuthology.orchestra.run.trial153.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.197 DEBUG:teuthology.orchestra.run.trial183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.233 INFO:teuthology.orchestra.run.trial183.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.242 DEBUG:teuthology.orchestra.run.trial191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-24T16:40:30.278 INFO:teuthology.orchestra.run.trial191.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEa3Hp6DP8aspGxXoeJ/i3C4TF2MV1u/tBoF7lOb1Etn ceph-6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:30.288 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2026-02-24T16:40:30.404 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:30.672 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2026-02-24T16:40:30.673 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch client-keyring set client.admin '*' --mode 0755 2026-02-24T16:40:30.791 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:30.894 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:30 trial151 ceph-mon[39453]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:30.894 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1888743477' entity='client.admin' 2026-02-24T16:40:31.043 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial051 2026-02-24T16:40:31.043 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:40:31.044 DEBUG:teuthology.orchestra.run.trial051:> dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:31.062 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:40:31.062 DEBUG:teuthology.orchestra.run.trial051:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:31.119 INFO:tasks.cephadm:Adding host trial051 to orchestrator... 2026-02-24T16:40:31.119 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host add trial051 2026-02-24T16:40:31.242 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial051", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial151", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:32.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:32 trial151 ceph-mon[39453]: Updating trial151:/etc/ceph/ceph.conf 2026-02-24T16:40:33.160 INFO:teuthology.orchestra.run.trial151.stdout:Added host 'trial051' with addr '10.20.193.51' 2026-02-24T16:40:33.202 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-24T16:40:33.320 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:33.435 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: Updating trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:33.435 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: Deploying cephadm binary to trial051 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: Updating trial151:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: Updating trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: Certificate for "cephadm-signed_agent_cert (trial151)" is still valid for 1824 days. 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:33.436 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:33.437 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:33.540 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:33.540 INFO:teuthology.orchestra.run.trial151.stdout:[{"addr": "10.20.193.151", "hostname": "trial151", "labels": [], "status": ""}, {"addr": "10.20.193.51", "hostname": "trial051", "labels": [], "status": ""}] 2026-02-24T16:40:33.575 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial052 2026-02-24T16:40:33.576 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:40:33.576 DEBUG:teuthology.orchestra.run.trial052:> dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:33.594 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:40:33.594 DEBUG:teuthology.orchestra.run.trial052:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:33.649 INFO:tasks.cephadm:Adding host trial052 to orchestrator... 2026-02-24T16:40:33.649 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host add trial052 2026-02-24T16:40:33.765 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:34.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:34 trial151 ceph-mon[39453]: Added host trial051 2026-02-24T16:40:34.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:34.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:34 trial151 ceph-mon[39453]: mgrmap e13: x(active, since 6s) 2026-02-24T16:40:34.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:34.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:34 trial151 ceph-mon[39453]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:40:35.562 INFO:teuthology.orchestra.run.trial151.stdout:Added host 'trial052' with addr '10.20.193.52' 2026-02-24T16:40:35.622 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-24T16:40:35.645 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:35 trial151 ceph-mon[39453]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial052", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:35.645 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:35.645 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:35 trial151 ceph-mon[39453]: Deploying cephadm binary to trial052 2026-02-24T16:40:35.736 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:35.952 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:35.952 INFO:teuthology.orchestra.run.trial151.stdout:[{"addr": "10.20.193.151", "hostname": "trial151", "labels": [], "status": ""}, {"addr": "10.20.193.51", "hostname": "trial051", "labels": [], "status": ""}, {"addr": "10.20.193.52", "hostname": "trial052", "labels": [], "status": ""}] 2026-02-24T16:40:35.989 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial153 2026-02-24T16:40:35.989 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:40:35.989 DEBUG:teuthology.orchestra.run.trial153:> dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:36.007 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:40:36.007 DEBUG:teuthology.orchestra.run.trial153:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:36.063 INFO:tasks.cephadm:Adding host trial153 to orchestrator... 2026-02-24T16:40:36.064 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host add trial153 2026-02-24T16:40:36.179 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial051", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:36.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: Updating trial051:/etc/ceph/ceph.conf 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: Added host trial052 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: Updating trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: Updating trial051:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:36.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:37.996 INFO:teuthology.orchestra.run.trial151.stdout:Added host 'trial153' with addr '10.20.193.153' 2026-02-24T16:40:38.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:37 trial151 ceph-mon[39453]: Updating trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:38.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:37 trial151 ceph-mon[39453]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:40:38.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:37 trial151 ceph-mon[39453]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial153", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:38.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.056 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-24T16:40:38.172 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:38.391 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:38.392 INFO:teuthology.orchestra.run.trial151.stdout:[{"addr": "10.20.193.151", "hostname": "trial151", "labels": [], "status": ""}, {"addr": "10.20.193.51", "hostname": "trial051", "labels": [], "status": ""}, {"addr": "10.20.193.52", "hostname": "trial052", "labels": [], "status": ""}, {"addr": "10.20.193.153", "hostname": "trial153", "labels": [], "status": ""}] 2026-02-24T16:40:38.429 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial183 2026-02-24T16:40:38.429 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:40:38.429 DEBUG:teuthology.orchestra.run.trial183:> dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:38.447 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:40:38.448 DEBUG:teuthology.orchestra.run.trial183:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:38.503 INFO:tasks.cephadm:Adding host trial183 to orchestrator... 2026-02-24T16:40:38.503 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host add trial183 2026-02-24T16:40:38.621 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:38.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: Deploying cephadm binary to trial153 2026-02-24T16:40:38.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial052", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:38.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: Updating trial052:/etc/ceph/ceph.conf 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: Updating trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: Added host trial153 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: Updating trial052:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: Updating trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:40:40.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.384 INFO:teuthology.orchestra.run.trial151.stdout:Added host 'trial183' with addr '10.20.193.183' 2026-02-24T16:40:40.424 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-24T16:40:40.541 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:40.757 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:40.758 INFO:teuthology.orchestra.run.trial151.stdout:[{"addr": "10.20.193.151", "hostname": "trial151", "labels": [], "status": ""}, {"addr": "10.20.193.51", "hostname": "trial051", "labels": [], "status": ""}, {"addr": "10.20.193.52", "hostname": "trial052", "labels": [], "status": ""}, {"addr": "10.20.193.153", "hostname": "trial153", "labels": [], "status": ""}, {"addr": "10.20.193.183", "hostname": "trial183", "labels": [], "status": ""}] 2026-02-24T16:40:40.795 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial191 2026-02-24T16:40:40.796 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:40:40.796 DEBUG:teuthology.orchestra.run.trial191:> dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:40.815 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:40:40.815 DEBUG:teuthology.orchestra.run.trial191:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:40.871 INFO:tasks.cephadm:Adding host trial191 to orchestrator... 2026-02-24T16:40:40.871 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host add trial191 2026-02-24T16:40:40.984 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial183", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: Deploying cephadm binary to trial183 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial153", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:40.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: Updating trial153:/etc/ceph/ceph.conf 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: Added host trial183 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: Updating trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: Updating trial153:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: Updating trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:41 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:42.787 INFO:teuthology.orchestra.run.trial151.stdout:Added host 'trial191' with addr '10.20.193.191' 2026-02-24T16:40:42.826 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-24T16:40:42.940 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial191", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: Deploying cephadm binary to trial191 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial183", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:43.054 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:43.055 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:42 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:43.155 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:40:43.155 INFO:teuthology.orchestra.run.trial151.stdout:[{"addr": "10.20.193.151", "hostname": "trial151", "labels": [], "status": ""}, {"addr": "10.20.193.51", "hostname": "trial051", "labels": [], "status": ""}, {"addr": "10.20.193.52", "hostname": "trial052", "labels": [], "status": ""}, {"addr": "10.20.193.153", "hostname": "trial153", "labels": [], "status": ""}, {"addr": "10.20.193.183", "hostname": "trial183", "labels": [], "status": ""}, {"addr": "10.20.193.191", "hostname": "trial191", "labels": [], "status": ""}] 2026-02-24T16:40:43.193 INFO:tasks.cephadm:Setting crush tunables to default 2026-02-24T16:40:43.193 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd crush tunables default 2026-02-24T16:40:43.309 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:44.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: Updating trial183:/etc/ceph/ceph.conf 2026-02-24T16:40:44.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: Added host trial191 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2346464497' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2026-02-24T16:40:44.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:43 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:44.456 INFO:teuthology.orchestra.run.trial151.stderr:adjusted tunables profile to default 2026-02-24T16:40:44.497 INFO:tasks.cephadm:Adding mon.a on trial151 2026-02-24T16:40:44.497 INFO:tasks.cephadm:Adding mon.b on trial153 2026-02-24T16:40:44.497 INFO:tasks.cephadm:Adding mon.c on trial183 2026-02-24T16:40:44.497 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch apply mon '3;trial151:10.20.193.151=a;trial153:10.20.193.153=b;trial183:10.20.193.183=c' 2026-02-24T16:40:44.613 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-24T16:40:44.614 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-24T16:40:44.837 INFO:teuthology.orchestra.run.trial191.stdout:Scheduled mon update... 2026-02-24T16:40:44.894 DEBUG:teuthology.orchestra.run.trial153:mon.b> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.b.service 2026-02-24T16:40:44.897 DEBUG:teuthology.orchestra.run.trial183:mon.c> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.c.service 2026-02-24T16:40:44.899 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2026-02-24T16:40:44.899 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph mon dump -f json 2026-02-24T16:40:45.056 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:45.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: Updating trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:45.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: Updating trial183:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:45.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: Updating trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:45.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:40:45.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:45.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2346464497' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2026-02-24T16:40:45.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:44 trial151 ceph-mon[39453]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:45.341 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:40:45.341 INFO:teuthology.orchestra.run.trial183.stdout:{"epoch":1,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","modified":"2026-02-24T16:40:05.376228Z","created":"2026-02-24T16:40:05.376228Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:3300","nonce":0},{"type":"v1","addr":"10.20.193.151:6789","nonce":0}]},"addr":"10.20.193.151:6789/0","public_addr":"10.20.193.151:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:05:376242+0000","crush_location":[]}],"quorum":[0]} 2026-02-24T16:40:45.342 INFO:teuthology.orchestra.run.trial183.stderr:dumped monmap epoch 1 2026-02-24T16:40:46.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;trial151:10.20.193.151=a;trial153:10.20.193.153=b;trial183:10.20.193.183=c", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:40:46.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Saving service mon spec with placement trial151:10.20.193.151=a;trial153:10.20.193.153=b;trial183:10.20.193.183=c;count:3 2026-02-24T16:40:46.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial191", "name": "osd_memory_target"} : dispatch 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Updating trial191:/etc/ceph/ceph.conf 2026-02-24T16:40:46.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Updating trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3823135246' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Updating trial191:/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Updating trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:46.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:45 trial151 ceph-mon[39453]: Deploying daemon mon.c on trial183 2026-02-24T16:40:46.377 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2026-02-24T16:40:46.378 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph mon dump -f json 2026-02-24T16:40:46.547 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:40:46.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:46 trial183 ceph-mon[43553]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2026-02-24T16:40:51.780 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:40:51.780 INFO:teuthology.orchestra.run.trial183.stdout:{"epoch":2,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","modified":"2026-02-24T16:40:46.773309Z","created":"2026-02-24T16:40:05.376228Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:3300","nonce":0},{"type":"v1","addr":"10.20.193.151:6789","nonce":0}]},"addr":"10.20.193.151:6789/0","public_addr":"10.20.193.151:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:05:376242+0000","crush_location":[]},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:3300","nonce":0},{"type":"v1","addr":"10.20.193.183:6789","nonce":0}]},"addr":"10.20.193.183:6789/0","public_addr":"10.20.193.183:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:46:773307+0000","crush_location":[]}],"quorum":[0,1]} 2026-02-24T16:40:51.780 INFO:teuthology.orchestra.run.trial183.stderr:dumped monmap epoch 2 2026-02-24T16:40:52.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: mon.a calling monitor election 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1097195501' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: mon.c calling monitor election 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: monmap epoch 2 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: last_changed 2026-02-24T16:40:46.773309+0000 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: min_mon_release 20 (tentacle) 2026-02-24T16:40:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: election_strategy: 1 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: 1: [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] mon.c 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: fsmap 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: mgrmap e13: x(active, since 25s) 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: overall HEALTH_OK 2026-02-24T16:40:52.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:51 trial183 ceph-mon[43553]: Deploying daemon mon.b on trial153 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: mon.a calling monitor election 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1097195501' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: mon.c calling monitor election 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: monmap epoch 2 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: last_changed 2026-02-24T16:40:46.773309+0000 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: min_mon_release 20 (tentacle) 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: election_strategy: 1 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:52.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: 1: [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] mon.c 2026-02-24T16:40:52.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: fsmap 2026-02-24T16:40:52.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:52.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: mgrmap e13: x(active, since 25s) 2026-02-24T16:40:52.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: overall HEALTH_OK 2026-02-24T16:40:52.239 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:51 trial151 ceph-mon[39453]: Deploying daemon mon.b on trial153 2026-02-24T16:40:52.825 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2026-02-24T16:40:52.825 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph mon dump -f json 2026-02-24T16:40:52.945 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:40:53.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:52 trial153 ceph-mon[44017]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2026-02-24T16:40:53.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:52 trial183 ceph-mon[43553]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:53.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:53.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:52 trial151 ceph-mon[39453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:53.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:58.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: mon.a calling monitor election 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: mon.c calling monitor election 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: mon.b calling monitor election 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: monmap epoch 3 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: last_changed 2026-02-24T16:40:52.980088+0000 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: min_mon_release 20 (tentacle) 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: election_strategy: 1 2026-02-24T16:40:58.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: 1: [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] mon.c 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: 2: [v2:10.20.193.153:3300/0,v1:10.20.193.153:6789/0] mon.b 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: fsmap 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: mgrmap e13: x(active, since 31s) 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: overall HEALTH_OK 2026-02-24T16:40:58.316 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:58 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:58.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: mon.a calling monitor election 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: mon.c calling monitor election 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: mon.b calling monitor election 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: monmap epoch 3 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: last_changed 2026-02-24T16:40:52.980088+0000 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: min_mon_release 20 (tentacle) 2026-02-24T16:40:58.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: election_strategy: 1 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: 1: [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] mon.c 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: 2: [v2:10.20.193.153:3300/0,v1:10.20.193.153:6789/0] mon.b 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: fsmap 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: mgrmap e13: x(active, since 31s) 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: overall HEALTH_OK 2026-02-24T16:40:58.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:58 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: mon.a calling monitor election 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: mon.c calling monitor election 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: mon.b calling monitor election 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: monmap epoch 3 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: fsid 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: last_changed 2026-02-24T16:40:52.980088+0000 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: created 2026-02-24T16:40:05.376228+0000 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: min_mon_release 20 (tentacle) 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: election_strategy: 1 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: 0: [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] mon.a 2026-02-24T16:40:58.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: 1: [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] mon.c 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: 2: [v2:10.20.193.153:3300/0,v1:10.20.193.153:6789/0] mon.b 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: fsmap 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: osdmap e4: 0 total, 0 up, 0 in 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: mgrmap e13: x(active, since 31s) 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: overall HEALTH_OK 2026-02-24T16:40:58.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:58 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.213 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:40:59.213 INFO:teuthology.orchestra.run.trial183.stdout:{"epoch":3,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","modified":"2026-02-24T16:40:52.980088Z","created":"2026-02-24T16:40:05.376228Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:3300","nonce":0},{"type":"v1","addr":"10.20.193.151:6789","nonce":0}]},"addr":"10.20.193.151:6789/0","public_addr":"10.20.193.151:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:05:376242+0000","crush_location":[]},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:3300","nonce":0},{"type":"v1","addr":"10.20.193.183:6789","nonce":0}]},"addr":"10.20.193.183:6789/0","public_addr":"10.20.193.183:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:46:773307+0000","crush_location":[]},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:3300","nonce":0},{"type":"v1","addr":"10.20.193.153:6789","nonce":0}]},"addr":"10.20.193.153:6789/0","public_addr":"10.20.193.153:6789/0","priority":0,"weight":0,"time_added":"2026-02-24T16:40:52:980086+0000","crush_location":[]}],"quorum":[0,1,2]} 2026-02-24T16:40:59.213 INFO:teuthology.orchestra.run.trial183.stderr:dumped monmap epoch 3 2026-02-24T16:40:59.253 INFO:tasks.cephadm:Generating final ceph.conf file... 2026-02-24T16:40:59.253 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph config generate-minimal-conf 2026-02-24T16:40:59.370 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial151:/etc/ceph/ceph.conf 2026-02-24T16:40:59.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial051:/etc/ceph/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial052:/etc/ceph/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial153:/etc/ceph/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial183:/etc/ceph/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial191:/etc/ceph/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Updating trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:59.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Reconfiguring mon.a (unknown last config time)... 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: Reconfiguring daemon mon.a on trial151 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.488 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.488 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.488 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:40:59 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2134764477' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial151:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial051:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial052:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial153:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial183:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial191:/etc/ceph/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Updating trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.566 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Reconfiguring mon.a (unknown last config time)... 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: Reconfiguring daemon mon.a on trial151 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.567 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.568 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:40:59 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2134764477' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:59.582 INFO:teuthology.orchestra.run.trial151.stdout:# minimal ceph.conf for 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:59.583 INFO:teuthology.orchestra.run.trial151.stdout:[global] 2026-02-24T16:40:59.583 INFO:teuthology.orchestra.run.trial151.stdout: fsid = 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:40:59.583 INFO:teuthology.orchestra.run.trial151.stdout: mon_host = [v2:10.20.193.151:3300/0,v1:10.20.193.151:6789/0] [v2:10.20.193.153:3300/0,v1:10.20.193.153:6789/0] [v2:10.20.193.183:3300/0,v1:10.20.193.183:6789/0] 2026-02-24T16:40:59.583 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial151:/etc/ceph/ceph.conf 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial051:/etc/ceph/ceph.conf 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial052:/etc/ceph/ceph.conf 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial153:/etc/ceph/ceph.conf 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial183:/etc/ceph/ceph.conf 2026-02-24T16:40:59.584 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial191:/etc/ceph/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Updating trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.585 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Reconfiguring mon.a (unknown last config time)... 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.586 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: Reconfiguring daemon mon.a on trial151 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:40:59.587 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:40:59 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2134764477' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-24T16:40:59.619 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2026-02-24T16:40:59.619 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:40:59.619 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:59.648 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:40:59.648 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:59.713 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:40:59.713 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:59.741 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:40:59.741 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:59.806 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:59.806 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:59.833 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:40:59.834 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:59.896 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:40:59.897 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:40:59.926 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:40:59.926 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:40:59.990 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:40:59.990 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:41:00.019 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:41:00.020 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:41:00.084 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:41:00.084 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/ceph/ceph.conf 2026-02-24T16:41:00.112 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:41:00.112 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-24T16:41:00.176 INFO:tasks.cephadm:Adding mgr.x on trial151 2026-02-24T16:41:00.176 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch apply mgr '1;trial151=x' 2026-02-24T16:41:00.332 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:00.567 INFO:teuthology.orchestra.run.trial191.stdout:Scheduled mgr update... 2026-02-24T16:41:00.626 DEBUG:tasks.cephadm:set 0 configs 2026-02-24T16:41:00.627 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph config dump 2026-02-24T16:41:00.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: Reconfiguring mon.b (monmap changed)... 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: Reconfiguring daemon mon.b on trial153 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: Reconfiguring mon.c (monmap changed)... 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: Reconfiguring daemon mon.c on trial183 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3220433172' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: Reconfiguring mon.b (monmap changed)... 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: Reconfiguring daemon mon.b on trial153 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: Reconfiguring mon.c (monmap changed)... 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: Reconfiguring daemon mon.c on trial183 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:00.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3220433172' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.742 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:00.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: Reconfiguring mon.b (monmap changed)... 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: Reconfiguring daemon mon.b on trial153 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: Reconfiguring mon.c (monmap changed)... 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: Reconfiguring daemon mon.c on trial183 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:00.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3220433172' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global dev auth_debug true 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global basic container_image quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18 * 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global dev debug_asserts_on_shutdown true 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global basic log_to_file true 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global basic log_to_journald false 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global basic log_to_stderr false 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_allow_pool_delete true 2026-02-24T16:41:00.964 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_clock_drift_allowed 1.000000 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_cluster_log_to_file true 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_max_pg_per_osd 10000 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global dev mon_warn_on_pool_no_app false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global advanced mon_warn_on_too_few_osds false 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global dev ms_die_on_bug true 2026-02-24T16:41:00.965 INFO:teuthology.orchestra.run.trial151.stdout:global dev ms_die_on_old_message true 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:global advanced osd_pool_default_erasure_code_profile plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:global advanced public_network 10.20.192.0/20 * 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced auth_service_ticket_ttl 240.000000 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced debug_mon 20/20 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced debug_ms 1/1 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced debug_paxos 20/20 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_data_avail_warn 5 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_down_mkfs_grace 300 2026-02-24T16:41:00.966 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_mgr_mkfs_grace 240 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon dev mon_osd_prime_pg_temp true 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_osd_reporter_subtree_level osd 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced debug_mgr 20/20 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced debug_ms 1/1 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mgr/cephadm/container_init True * 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mgr/cephadm/migration_current 8 * 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2026-02-24T16:41:00.967 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev bdev_debug_aio true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced debug_ms 1/1 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced debug_osd 20/20 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_misdirected_ops true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_op_order true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_pg_log_writeout true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_shutdown true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_verify_cached_snaps true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_verify_missing_on_start true 2026-02-24T16:41:00.968 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_debug_verify_stray_on_activate true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_mclock_profile high_recovery_ops 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd dev osd_mclock_skip_benchmark true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_memory_target_autotune true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_op_queue debug_random * 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_op_queue_cut_off debug_random * 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_recover_clone_overlap true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_recovery_max_chunk 1048576 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_scrub_load_threshold 5.000000 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_scrub_max_interval 600.000000 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:osd advanced osd_shutdown_pgref_assert true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:client.rgw advanced rgw_cache_enabled true 2026-02-24T16:41:00.969 INFO:teuthology.orchestra.run.trial151.stdout:client.rgw advanced rgw_enable_ops_log true 2026-02-24T16:41:00.970 INFO:teuthology.orchestra.run.trial151.stdout:client.rgw advanced rgw_enable_usage_log true 2026-02-24T16:41:01.001 INFO:tasks.cephadm:Deploying OSDs... 2026-02-24T16:41:01.002 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:41:01.002 DEBUG:teuthology.orchestra.run.trial051:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:01.020 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:01.021 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-24 16:40:35.197212760 +0000 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.077 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.078 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:01.139 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-24T16:41:01.139 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-24T16:41:01.139 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000149127 s, 3.4 MB/s 2026-02-24T16:41:01.141 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:01.198 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:01.254 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:01.254 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.254 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-24T16:41:01.254 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.255 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.255 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-24 16:40:35.228212270 +0000 2026-02-24T16:41:01.255 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-24 16:37:59.201127388 +0000 2026-02-24T16:41:01.255 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-24 16:37:59.201127388 +0000 2026-02-24T16:41:01.255 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-24 16:37:59.201127388 +0000 2026-02-24T16:41:01.255 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:01.317 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-24T16:41:01.317 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-24T16:41:01.318 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000149097 s, 3.4 MB/s 2026-02-24T16:41:01.319 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:01.376 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:01.432 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:01.432 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1296 Links: 1 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-24 16:40:35.259211779 +0000 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.433 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.434 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:01.496 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-24T16:41:01.496 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-24T16:41:01.496 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000152744 s, 3.4 MB/s 2026-02-24T16:41:01.497 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:01.554 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-24 16:40:35.289211305 +0000 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.611 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.612 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-24 16:37:59.200127443 +0000 2026-02-24T16:41:01.612 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:01.672 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-24T16:41:01.672 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-24T16:41:01.672 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000141483 s, 3.6 MB/s 2026-02-24T16:41:01.674 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:01.730 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:41:01.730 DEBUG:teuthology.orchestra.run.trial052:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:01.749 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:01.749 DEBUG:teuthology.orchestra.run.trial052:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout:Access: 2026-02-24 16:40:37.561854024 +0000 2026-02-24T16:41:01.806 INFO:teuthology.orchestra.run.trial052.stdout:Modify: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.807 INFO:teuthology.orchestra.run.trial052.stdout:Change: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.807 INFO:teuthology.orchestra.run.trial052.stdout: Birth: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.807 DEBUG:teuthology.orchestra.run.trial052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:01.869 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records in 2026-02-24T16:41:01.869 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records out 2026-02-24T16:41:01.869 INFO:teuthology.orchestra.run.trial052.stderr:512 bytes copied, 0.000167653 s, 3.1 MB/s 2026-02-24T16:41:01.870 DEBUG:teuthology.orchestra.run.trial052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:01.928 DEBUG:teuthology.orchestra.run.trial052:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;trial151=x", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: Saving service mgr spec with placement trial151=x;count:1 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: Reconfiguring mgr.x (unknown last config time)... 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: Reconfiguring daemon mgr.x on trial151 2026-02-24T16:41:01.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1170807070' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2026-02-24T16:41:01.986 INFO:teuthology.orchestra.run.trial052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:01.986 INFO:teuthology.orchestra.run.trial052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:01.986 INFO:teuthology.orchestra.run.trial052.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-24T16:41:01.986 INFO:teuthology.orchestra.run.trial052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:01.986 INFO:teuthology.orchestra.run.trial052.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:01.987 INFO:teuthology.orchestra.run.trial052.stdout:Access: 2026-02-24 16:40:37.593854391 +0000 2026-02-24T16:41:01.987 INFO:teuthology.orchestra.run.trial052.stdout:Modify: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.987 INFO:teuthology.orchestra.run.trial052.stdout:Change: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.987 INFO:teuthology.orchestra.run.trial052.stdout: Birth: 2026-02-24 16:37:59.313660831 +0000 2026-02-24T16:41:01.987 DEBUG:teuthology.orchestra.run.trial052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:01.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:01.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;trial151=x", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:01.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: Saving service mgr spec with placement trial151=x;count:1 2026-02-24T16:41:01.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: Reconfiguring mgr.x (unknown last config time)... 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: Reconfiguring daemon mgr.x on trial151 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:01.989 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1170807070' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2026-02-24T16:41:02.049 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records in 2026-02-24T16:41:02.049 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records out 2026-02-24T16:41:02.049 INFO:teuthology.orchestra.run.trial052.stderr:512 bytes copied, 0.000151593 s, 3.4 MB/s 2026-02-24T16:41:02.051 DEBUG:teuthology.orchestra.run.trial052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;trial151=x", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: Saving service mgr spec with placement trial151=x;count:1 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: Reconfiguring mgr.x (unknown last config time)... 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: Reconfiguring daemon mgr.x on trial151 2026-02-24T16:41:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:02.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:02.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1170807070' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2026-02-24T16:41:02.108 DEBUG:teuthology.orchestra.run.trial052:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Access: 2026-02-24 16:40:37.624854746 +0000 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Modify: 2026-02-24 16:37:59.314660810 +0000 2026-02-24T16:41:02.166 INFO:teuthology.orchestra.run.trial052.stdout:Change: 2026-02-24 16:37:59.314660810 +0000 2026-02-24T16:41:02.167 INFO:teuthology.orchestra.run.trial052.stdout: Birth: 2026-02-24 16:37:59.314660810 +0000 2026-02-24T16:41:02.167 DEBUG:teuthology.orchestra.run.trial052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:02.229 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records in 2026-02-24T16:41:02.230 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records out 2026-02-24T16:41:02.230 INFO:teuthology.orchestra.run.trial052.stderr:512 bytes copied, 0.000145131 s, 3.5 MB/s 2026-02-24T16:41:02.233 DEBUG:teuthology.orchestra.run.trial052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:02.290 DEBUG:teuthology.orchestra.run.trial052:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:02.347 INFO:teuthology.orchestra.run.trial052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:02.347 INFO:teuthology.orchestra.run.trial052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:02.347 INFO:teuthology.orchestra.run.trial052.stdout:Device: 6h/6d Inode: 1283 Links: 1 2026-02-24T16:41:02.347 INFO:teuthology.orchestra.run.trial052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:02.348 INFO:teuthology.orchestra.run.trial052.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:02.348 INFO:teuthology.orchestra.run.trial052.stdout:Access: 2026-02-24 16:40:37.654855090 +0000 2026-02-24T16:41:02.348 INFO:teuthology.orchestra.run.trial052.stdout:Modify: 2026-02-24 16:37:59.312660852 +0000 2026-02-24T16:41:02.348 INFO:teuthology.orchestra.run.trial052.stdout:Change: 2026-02-24 16:37:59.312660852 +0000 2026-02-24T16:41:02.348 INFO:teuthology.orchestra.run.trial052.stdout: Birth: 2026-02-24 16:37:59.312660852 +0000 2026-02-24T16:41:02.348 DEBUG:teuthology.orchestra.run.trial052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:02.410 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records in 2026-02-24T16:41:02.410 INFO:teuthology.orchestra.run.trial052.stderr:1+0 records out 2026-02-24T16:41:02.411 INFO:teuthology.orchestra.run.trial052.stderr:512 bytes copied, 0.000153156 s, 3.3 MB/s 2026-02-24T16:41:02.412 DEBUG:teuthology.orchestra.run.trial052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:02.469 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:41:02.469 DEBUG:teuthology.orchestra.run.trial151:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:02.489 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:02.490 DEBUG:teuthology.orchestra.run.trial151:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:02.546 INFO:teuthology.orchestra.run.trial151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Device: 6h/6d Inode: 1311 Links: 1 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Access: 2026-02-24 16:40:31.377727139 +0000 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Modify: 2026-02-24 16:38:28.380231784 +0000 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout:Change: 2026-02-24 16:38:28.380231784 +0000 2026-02-24T16:41:02.547 INFO:teuthology.orchestra.run.trial151.stdout: Birth: 2026-02-24 16:38:28.380231784 +0000 2026-02-24T16:41:02.548 DEBUG:teuthology.orchestra.run.trial151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:02.611 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records in 2026-02-24T16:41:02.611 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records out 2026-02-24T16:41:02.611 INFO:teuthology.orchestra.run.trial151.stderr:512 bytes copied, 0.000152046 s, 3.4 MB/s 2026-02-24T16:41:02.612 DEBUG:teuthology.orchestra.run.trial151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:02.670 DEBUG:teuthology.orchestra.run.trial151:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:02.726 INFO:teuthology.orchestra.run.trial151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Access: 2026-02-24 16:40:31.414728113 +0000 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Modify: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout:Change: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.727 INFO:teuthology.orchestra.run.trial151.stdout: Birth: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.728 DEBUG:teuthology.orchestra.run.trial151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:02.789 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records in 2026-02-24T16:41:02.790 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records out 2026-02-24T16:41:02.790 INFO:teuthology.orchestra.run.trial151.stderr:512 bytes copied, 0.000140805 s, 3.6 MB/s 2026-02-24T16:41:02.791 DEBUG:teuthology.orchestra.run.trial151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:02.849 DEBUG:teuthology.orchestra.run.trial151:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:02.906 INFO:teuthology.orchestra.run.trial151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:02.906 INFO:teuthology.orchestra.run.trial151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:02.906 INFO:teuthology.orchestra.run.trial151.stdout:Device: 6h/6d Inode: 1300 Links: 1 2026-02-24T16:41:02.906 INFO:teuthology.orchestra.run.trial151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:02.907 INFO:teuthology.orchestra.run.trial151.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:02.907 INFO:teuthology.orchestra.run.trial151.stdout:Access: 2026-02-24 16:40:31.449729035 +0000 2026-02-24T16:41:02.907 INFO:teuthology.orchestra.run.trial151.stdout:Modify: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.907 INFO:teuthology.orchestra.run.trial151.stdout:Change: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.907 INFO:teuthology.orchestra.run.trial151.stdout: Birth: 2026-02-24 16:38:28.379231788 +0000 2026-02-24T16:41:02.907 DEBUG:teuthology.orchestra.run.trial151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:02.969 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records in 2026-02-24T16:41:02.970 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records out 2026-02-24T16:41:02.970 INFO:teuthology.orchestra.run.trial151.stderr:512 bytes copied, 0.000168678 s, 3.0 MB/s 2026-02-24T16:41:02.971 DEBUG:teuthology.orchestra.run.trial151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:03.029 DEBUG:teuthology.orchestra.run.trial151:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Device: 6h/6d Inode: 1288 Links: 1 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Access: 2026-02-24 16:40:31.483729930 +0000 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Modify: 2026-02-24 16:38:28.378231793 +0000 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout:Change: 2026-02-24 16:38:28.378231793 +0000 2026-02-24T16:41:03.085 INFO:teuthology.orchestra.run.trial151.stdout: Birth: 2026-02-24 16:38:28.378231793 +0000 2026-02-24T16:41:03.086 DEBUG:teuthology.orchestra.run.trial151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:03.148 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records in 2026-02-24T16:41:03.149 INFO:teuthology.orchestra.run.trial151.stderr:1+0 records out 2026-02-24T16:41:03.149 INFO:teuthology.orchestra.run.trial151.stderr:512 bytes copied, 0.000147909 s, 3.5 MB/s 2026-02-24T16:41:03.150 DEBUG:teuthology.orchestra.run.trial151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:03.207 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:41:03.207 DEBUG:teuthology.orchestra.run.trial153:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:03.226 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:03.226 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:03.282 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1296 Links: 1 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-24 16:40:52.766298215 +0000 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.283 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.284 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:03.347 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-24T16:41:03.347 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-24T16:41:03.347 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000161362 s, 3.2 MB/s 2026-02-24T16:41:03.348 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:03.406 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:03.462 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:03.462 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:03.462 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-24T16:41:03.462 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:03.463 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:03.463 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-24 16:40:52.766298215 +0000 2026-02-24T16:41:03.463 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-24 16:38:06.280809574 +0000 2026-02-24T16:41:03.463 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-24 16:38:06.280809574 +0000 2026-02-24T16:41:03.463 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-24 16:38:06.280809574 +0000 2026-02-24T16:41:03.463 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:03.524 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-24T16:41:03.524 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-24T16:41:03.525 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000151554 s, 3.4 MB/s 2026-02-24T16:41:03.526 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:03.583 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:03.639 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-24 16:40:52.767298200 +0000 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.640 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.641 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:03.703 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-24T16:41:03.703 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-24T16:41:03.703 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000150661 s, 3.4 MB/s 2026-02-24T16:41:03.704 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:03.762 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:03.818 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1289 Links: 1 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-24 16:40:52.767298200 +0000 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.819 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-24 16:38:06.279809589 +0000 2026-02-24T16:41:03.820 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:03.883 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-24T16:41:03.883 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-24T16:41:03.884 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000142647 s, 3.6 MB/s 2026-02-24T16:41:03.885 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:03.942 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:41:03.943 DEBUG:teuthology.orchestra.run.trial183:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:03.961 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:03.961 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:04.017 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:04.017 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1308 Links: 1 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-24 16:40:46.556941625 +0000 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.018 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.019 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:03 trial153 ceph-mon[44017]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:04.078 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:03 trial183 ceph-mon[43553]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:04.080 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-24T16:41:04.080 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-24T16:41:04.080 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000170106 s, 3.0 MB/s 2026-02-24T16:41:04.081 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:04.139 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:04.194 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:04.194 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-24 16:40:46.556941625 +0000 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.195 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.195 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:04.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:03 trial151 ceph-mon[39453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:04.257 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-24T16:41:04.257 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-24T16:41:04.258 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000149418 s, 3.4 MB/s 2026-02-24T16:41:04.259 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:04.317 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:04.373 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:04.373 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.373 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-24T16:41:04.373 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.373 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.374 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-24 16:40:46.557941597 +0000 2026-02-24T16:41:04.374 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.374 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.374 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-24 16:38:24.023821795 +0000 2026-02-24T16:41:04.374 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:04.436 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-24T16:41:04.436 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-24T16:41:04.436 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000157002 s, 3.3 MB/s 2026-02-24T16:41:04.437 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:04.494 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-24 16:40:46.557941597 +0000 2026-02-24T16:41:04.551 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-24 16:38:24.022821819 +0000 2026-02-24T16:41:04.552 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-24 16:38:24.022821819 +0000 2026-02-24T16:41:04.552 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-24 16:38:24.022821819 +0000 2026-02-24T16:41:04.552 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:04.614 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-24T16:41:04.614 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-24T16:41:04.614 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000146903 s, 3.5 MB/s 2026-02-24T16:41:04.615 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:04.673 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:41:04.673 DEBUG:teuthology.orchestra.run.trial191:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:41:04.692 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:41:04.692 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_1 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 1319 Links: 1 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-24 16:40:44.836236480 +0000 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-24 16:37:56.749756211 +0000 2026-02-24T16:41:04.748 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-24 16:37:56.749756211 +0000 2026-02-24T16:41:04.749 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-24 16:37:56.749756211 +0000 2026-02-24T16:41:04.749 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:41:04.812 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-24T16:41:04.812 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-24T16:41:04.812 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000177142 s, 2.9 MB/s 2026-02-24T16:41:04.814 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:41:04.871 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_2 2026-02-24T16:41:04.927 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:41:04.927 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:04.927 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 1313 Links: 1 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-24 16:40:44.869236178 +0000 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-24 16:37:56.748756220 +0000 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-24 16:37:56.748756220 +0000 2026-02-24T16:41:04.928 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-24 16:37:56.748756220 +0000 2026-02-24T16:41:04.928 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:41:04.991 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-24T16:41:04.991 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-24T16:41:04.991 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000140473 s, 3.6 MB/s 2026-02-24T16:41:04.992 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:41:05.051 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_3 2026-02-24T16:41:05.107 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-24 16:40:44.904235858 +0000 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.108 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.109 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:41:05.170 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-24T16:41:05.170 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-24T16:41:05.170 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000147446 s, 3.5 MB/s 2026-02-24T16:41:05.172 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:41:05.229 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_4 2026-02-24T16:41:05.284 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-24 16:40:44.937235556 +0000 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.285 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-24 16:37:56.747756229 +0000 2026-02-24T16:41:05.286 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:41:05.347 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-24T16:41:05.347 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-24T16:41:05.347 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000145112 s, 3.5 MB/s 2026-02-24T16:41:05.349 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:41:05.406 INFO:tasks.cephadm:Deploying osd.0 on trial151 with /dev/vg_nvme/lv_4... 2026-02-24T16:41:05.406 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-24T16:41:05.525 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:05.935 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:05 trial151 ceph-mon[39453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:05.974 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:05.992 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch daemon add osd trial151:vg_nvme/lv_4 --skip-validation 2026-02-24T16:41:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:05 trial153 ceph-mon[44017]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:06.109 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:06.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:05 trial183 ceph-mon[43553]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:07.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial151:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:07.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: osd.default does not exist. Creating it now. 2026-02-24T16:41:07.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: Creating OSDs with service ID: default on trial151:['vg_nvme/lv_4'] 2026-02-24T16:41:07.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: Marking host: trial151 for OSDSpec preview refresh. 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: Saving service osd.default spec with placement trial151 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"} : dispatch 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"}]': finished 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: osdmap e5: 1 total, 0 up, 1 in 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:07.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:07 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1217646793' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:07.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial151:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: osd.default does not exist. Creating it now. 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: Creating OSDs with service ID: default on trial151:['vg_nvme/lv_4'] 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: Marking host: trial151 for OSDSpec preview refresh. 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: Saving service osd.default spec with placement trial151 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"} : dispatch 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"}]': finished 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: osdmap e5: 1 total, 0 up, 1 in 2026-02-24T16:41:07.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:07.736 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:07 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1217646793' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:07.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial151:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:07.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: osd.default does not exist. Creating it now. 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: Creating OSDs with service ID: default on trial151:['vg_nvme/lv_4'] 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: Marking host: trial151 for OSDSpec preview refresh. 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: Saving service osd.default spec with placement trial151 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/656757484' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ba569b-fd87-4dcc-8ca5-fa963049100c"}]': finished 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: osdmap e5: 1 total, 0 up, 1 in 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:07.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:07 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1217646793' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:09.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:09 trial151 ceph-mon[39453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:09.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:09 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:09.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:09 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:09.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:09 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:09.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:09 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:09 trial183 ceph-mon[43553]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:09 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:09 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:09 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:09 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:09 trial153 ceph-mon[44017]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:09 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:09 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:09 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-24T16:41:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:09 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:10.915 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:10 trial151 ceph-mon[39453]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:10.915 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:10 trial151 ceph-mon[39453]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:10.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:10 trial183 ceph-mon[43553]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:10.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:10 trial183 ceph-mon[43553]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:11.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:10 trial153 ceph-mon[44017]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:11.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:10 trial153 ceph-mon[44017]: Deploying daemon osd.0 on trial151 2026-02-24T16:41:11.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:11 trial151 ceph-mon[39453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:11.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:11 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:11.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:11 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:11 trial183 ceph-mon[43553]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:11 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:11 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:11 trial153 ceph-mon[44017]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:11 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:11 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2026-02-24T16:41:13.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:13 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2026-02-24T16:41:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:13 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:14.060 INFO:teuthology.orchestra.run.trial151.stdout:Created osd(s) 0 on host 'trial151' 2026-02-24T16:41:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2026-02-24T16:41:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:13 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:14.103 DEBUG:teuthology.orchestra.run.trial151:osd.0> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.0.service 2026-02-24T16:41:14.106 INFO:tasks.cephadm:Deploying osd.1 on trial153 with /dev/vg_nvme/lv_4... 2026-02-24T16:41:14.106 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-24T16:41:14.223 INFO:teuthology.orchestra.run.trial153.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.b/config 2026-02-24T16:41:14.698 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-24T16:41:14.716 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch daemon add osd trial153:vg_nvme/lv_4 --skip-validation 2026-02-24T16:41:14.833 INFO:teuthology.orchestra.run.trial153.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.b/config 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: osdmap e6: 1 total, 0 up, 1 in 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]} : dispatch 2026-02-24T16:41:15.207 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:15.208 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:14 trial153 ceph-mon[44017]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:15.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: osdmap e6: 1 total, 0 up, 1 in 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]} : dispatch 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:15.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:14 trial183 ceph-mon[43553]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:15.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: osdmap e6: 1 total, 0 up, 1 in 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]} : dispatch 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:15.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:14 trial151 ceph-mon[39453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-24T16:41:15.615 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0[57357]: 2026-02-24T16:41:15.483+0000 7f232b941640 -1 osd.0 0 waiting for initial osdmap 2026-02-24T16:41:15.615 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0[57357]: 2026-02-24T16:41:15.485+0000 7f2323800640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T16:41:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: Detected new or changed devices on trial151 2026-02-24T16:41:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: Adjusting osd_memory_target on trial151 to 84047M 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='client.24128 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial153:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:16.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: osd.default does not exist. Creating it now. 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: Creating OSDs with service ID: default on trial153:['vg_nvme/lv_4'] 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: Marking host: trial153 for OSDSpec preview refresh. 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: Saving service osd.default spec with placement trial153 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]}]': finished 2026-02-24T16:41:16.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: osdmap e7: 1 total, 0 up, 1 in 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"}]': finished 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519] boot 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: osdmap e8: 2 total, 1 up, 2 in 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:16.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4200929939' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:16.186 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: Detected new or changed devices on trial151 2026-02-24T16:41:16.186 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: Adjusting osd_memory_target on trial151 to 84047M 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.187 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='client.24128 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial153:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: osd.default does not exist. Creating it now. 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: Creating OSDs with service ID: default on trial153:['vg_nvme/lv_4'] 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: Marking host: trial153 for OSDSpec preview refresh. 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: Saving service osd.default spec with placement trial153 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]}]': finished 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: osdmap e7: 1 total, 0 up, 1 in 2026-02-24T16:41:16.188 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"}]': finished 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519] boot 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: osdmap e8: 2 total, 1 up, 2 in 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:16.189 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4200929939' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: Detected new or changed devices on trial151 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: Adjusting osd_memory_target on trial151 to 84047M 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='client.24128 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial153:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: osd.default does not exist. Creating it now. 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: Creating OSDs with service ID: default on trial153:['vg_nvme/lv_4'] 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: Marking host: trial153 for OSDSpec preview refresh. 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: Saving service osd.default spec with placement trial153 2026-02-24T16:41:16.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial151", "root=default"]}]': finished 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: osdmap e7: 1 total, 0 up, 1 in 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"}]': finished 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: osd.0 [v2:10.20.193.151:6802/110062519,v1:10.20.193.151:6803/110062519] boot 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: osdmap e8: 2 total, 1 up, 2 in 2026-02-24T16:41:16.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-24T16:41:16.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:16.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4200929939' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6cb9011d-41ad-4cf7-94b0-d899df29dfc3"} : dispatch 2026-02-24T16:41:17.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: purged_snaps scrub starts 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: purged_snaps scrub ok 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4254232697' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: pgmap v23: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:16 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: purged_snaps scrub starts 2026-02-24T16:41:17.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: purged_snaps scrub ok 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4254232697' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: pgmap v23: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:17 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: purged_snaps scrub starts 2026-02-24T16:41:17.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: purged_snaps scrub ok 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4254232697' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: pgmap v23: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:17.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:17 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.159 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: osdmap e9: 2 total, 1 up, 2 in 2026-02-24T16:41:18.159 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:18.159 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.159 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.159 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2026-02-24T16:41:18.160 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:18.160 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:18 trial153 ceph-mon[44017]: Deploying daemon osd.1 on trial153 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: osdmap e9: 2 total, 1 up, 2 in 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:18.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:18 trial183 ceph-mon[43553]: Deploying daemon osd.1 on trial153 2026-02-24T16:41:18.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: osdmap e9: 2 total, 1 up, 2 in 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:18.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:17 trial151 ceph-mon[39453]: Deploying daemon osd.1 on trial153 2026-02-24T16:41:19.182 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.182 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.183 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:19 trial153 ceph-mon[44017]: pgmap v25: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:19.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:19 trial183 ceph-mon[43553]: pgmap v25: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:19.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:19.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:19 trial151 ceph-mon[39453]: pgmap v25: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:20.083 INFO:teuthology.orchestra.run.trial153.stdout:Created osd(s) 1 on host 'trial153' 2026-02-24T16:41:20.124 DEBUG:teuthology.orchestra.run.trial153:osd.1> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.1.service 2026-02-24T16:41:20.126 INFO:tasks.cephadm:Deploying osd.2 on trial183 with /dev/vg_nvme/lv_4... 2026-02-24T16:41:20.126 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-24T16:41:20.244 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:20 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.400 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:20 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:20 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:20.722 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-24T16:41:20.740 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch daemon add osd trial183:vg_nvme/lv_4 --skip-validation 2026-02-24T16:41:20.857 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:41:21.276 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: osdmap e10: 2 total, 1 up, 2 in 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: pgmap v27: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.277 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.278 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:21 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.278 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1[47523]: 2026-02-24T16:41:21.233+0000 7f9c618ad640 -1 osd.1 0 waiting for initial osdmap 2026-02-24T16:41:21.279 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1[47523]: 2026-02-24T16:41:21.235+0000 7f9c5976c640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T16:41:21.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: osdmap e10: 2 total, 1 up, 2 in 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: pgmap v27: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:21 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: osdmap e10: 2 total, 1 up, 2 in 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]} : dispatch 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: pgmap v27: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:21.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:21 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial183:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: osd.default does not exist. Creating it now. 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: Creating OSDs with service ID: default on trial183:['vg_nvme/lv_4'] 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: Marking host: trial183 for OSDSpec preview refresh. 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: Saving service osd.default spec with placement trial183 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]}]': finished 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: osdmap e11: 2 total, 1 up, 2 in 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"}]': finished 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599] boot 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: osdmap e12: 3 total, 2 up, 3 in 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2354186516' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2647070972' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial183:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: osd.default does not exist. Creating it now. 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: Creating OSDs with service ID: default on trial183:['vg_nvme/lv_4'] 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: Marking host: trial183 for OSDSpec preview refresh. 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: Saving service osd.default spec with placement trial183 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]}]': finished 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: osdmap e11: 2 total, 1 up, 2 in 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"}]': finished 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599] boot 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: osdmap e12: 3 total, 2 up, 3 in 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2354186516' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2647070972' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:22.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial183:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: osd.default does not exist. Creating it now. 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: Creating OSDs with service ID: default on trial183:['vg_nvme/lv_4'] 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: Marking host: trial183 for OSDSpec preview refresh. 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: Saving service osd.default spec with placement trial183 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial153", "root=default"]}]': finished 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: osdmap e11: 2 total, 1 up, 2 in 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"}]': finished 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: osd.1 [v2:10.20.193.153:6800/3548290599,v1:10.20.193.153:6801/3548290599] boot 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: osdmap e12: 3 total, 2 up, 3 in 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2354186516' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ed27ee6a-67e6-48f8-a96b-9f87ce25870d"} : dispatch 2026-02-24T16:41:22.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2647070972' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:23.589 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: purged_snaps scrub starts 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: purged_snaps scrub ok 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: Detected new or changed devices on trial153 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: Adjusting osd_memory_target on trial153 to 88143M 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:23.590 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: pgmap v30: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: osdmap e13: 3 total, 2 up, 3 in 2026-02-24T16:41:23.591 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:23 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:23.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: purged_snaps scrub starts 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: purged_snaps scrub ok 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: Detected new or changed devices on trial153 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: Adjusting osd_memory_target on trial153 to 88143M 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:23.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: pgmap v30: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: osdmap e13: 3 total, 2 up, 3 in 2026-02-24T16:41:23.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:23 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:23.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: purged_snaps scrub starts 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: purged_snaps scrub ok 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: Detected new or changed devices on trial153 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: Adjusting osd_memory_target on trial153 to 88143M 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: pgmap v30: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: osdmap e13: 3 total, 2 up, 3 in 2026-02-24T16:41:23.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:23 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:24.691 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:24 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.691 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:24 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:24.691 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:24 trial183 ceph-mon[43553]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:24.691 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:24 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.691 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:24 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:24.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:24 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:24 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:24.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:24 trial153 ceph-mon[44017]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:24.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:24 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:24 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:24.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:24 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:24 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:24.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:24 trial151 ceph-mon[39453]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:24.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:24 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-24T16:41:24.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:24 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:25.528 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:25 trial183 ceph-mon[43553]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:25.529 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:25 trial183 ceph-mon[43553]: pgmap v32: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:25.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:25 trial153 ceph-mon[44017]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:25.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:25 trial153 ceph-mon[44017]: pgmap v32: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:25.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:25 trial151 ceph-mon[39453]: Deploying daemon osd.2 on trial183 2026-02-24T16:41:25.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:25 trial151 ceph-mon[39453]: pgmap v32: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:26.683 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:26 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:26.683 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:26 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:26.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:26 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:26.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:26 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:26.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:26.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:26 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:27.760 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:27 trial183 ceph-mon[43553]: pgmap v33: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:27.760 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:27 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:27.760 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:27 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:27.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:27 trial151 ceph-mon[39453]: pgmap v33: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:27.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:27.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:27 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:27 trial153 ceph-mon[44017]: pgmap v33: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:27 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:27 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:28.102 INFO:teuthology.orchestra.run.trial183.stdout:Created osd(s) 2 on host 'trial183' 2026-02-24T16:41:28.143 DEBUG:teuthology.orchestra.run.trial183:osd.2> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.2.service 2026-02-24T16:41:28.145 INFO:tasks.cephadm:Deploying osd.3 on trial191 with /dev/vg_nvme/lv_4... 2026-02-24T16:41:28.145 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-24T16:41:28.262 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:28.484 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:41:28.212+0000 7f066f0bc780 -1 osd.2 0 log_to_monitors true 2026-02-24T16:41:28.764 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-24T16:41:28.780 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch daemon add osd trial191:vg_nvme/lv_4 --skip-validation 2026-02-24T16:41:28.947 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:29.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: pgmap v34: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:29 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:29.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: pgmap v34: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:29 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:29.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-24T16:41:29.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: pgmap v34: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:29.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:29.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:29 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:30.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:41:29.864+0000 7f066b03e640 -1 osd.2 0 waiting for initial osdmap 2026-02-24T16:41:30.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:41:29.866+0000 7f0666f05640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: osdmap e14: 3 total, 2 up, 3 in 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial191:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: osd.default does not exist. Creating it now. 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: Creating OSDs with service ID: default on trial191:['vg_nvme/lv_4'] 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: Marking host: trial191 for OSDSpec preview refresh. 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: Saving service osd.default spec with placement trial191 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:30.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"} : dispatch 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]}]': finished 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"}]': finished 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: osdmap e15: 4 total, 2 up, 4 in 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:30.237 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:30 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: osdmap e14: 3 total, 2 up, 3 in 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial191:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: osd.default does not exist. Creating it now. 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: Creating OSDs with service ID: default on trial191:['vg_nvme/lv_4'] 2026-02-24T16:41:30.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: Marking host: trial191 for OSDSpec preview refresh. 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: Saving service osd.default spec with placement trial191 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"} : dispatch 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]}]': finished 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"}]': finished 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: osdmap e15: 4 total, 2 up, 4 in 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:30.488 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:30 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-24T16:41:30.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: osdmap e14: 3 total, 2 up, 3 in 2026-02-24T16:41:30.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]} : dispatch 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial191:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: osd.default does not exist. Creating it now. 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: Creating OSDs with service ID: default on trial191:['vg_nvme/lv_4'] 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: Marking host: trial191 for OSDSpec preview refresh. 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: Saving service osd.default spec with placement trial191 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:30.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"} : dispatch 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial183", "root=default"]}]': finished 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/447831231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef31c97-bc0f-41bd-bca9-d13596a02d4a"}]': finished 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: osdmap e15: 4 total, 2 up, 4 in 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:30.565 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:30 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: Detected new or changed devices on trial183 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: Adjusting osd_memory_target on trial183 to 88143M 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/285116997' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:31.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: pgmap v37: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:31.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410] boot 2026-02-24T16:41:31.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: osdmap e16: 4 total, 3 up, 4 in 2026-02-24T16:41:31.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:31.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:31 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: Detected new or changed devices on trial183 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: Adjusting osd_memory_target on trial183 to 88143M 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/285116997' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: pgmap v37: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410] boot 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: osdmap e16: 4 total, 3 up, 4 in 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:31.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:31 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: Detected new or changed devices on trial183 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: Adjusting osd_memory_target on trial183 to 88143M 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/285116997' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-24T16:41:31.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: pgmap v37: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-24T16:41:31.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: osd.2 [v2:10.20.193.183:6800/1741249410,v1:10.20.193.183:6801/1741249410] boot 2026-02-24T16:41:31.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: osdmap e16: 4 total, 3 up, 4 in 2026-02-24T16:41:31.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-24T16:41:31.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:31 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:32.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: purged_snaps scrub starts 2026-02-24T16:41:32.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: purged_snaps scrub ok 2026-02-24T16:41:32.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:32.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:32.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: osdmap e17: 4 total, 3 up, 4 in 2026-02-24T16:41:32.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:32 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: purged_snaps scrub starts 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: purged_snaps scrub ok 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: osdmap e17: 4 total, 3 up, 4 in 2026-02-24T16:41:32.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:32 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: purged_snaps scrub starts 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: purged_snaps scrub ok 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: osdmap e17: 4 total, 3 up, 4 in 2026-02-24T16:41:32.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:32 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:33.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:33.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:33.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: pgmap v40: 0 pgs: ; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:33.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: pgmap v40: 0 pgs: ; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:33.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: Deploying daemon osd.3 on trial191 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: pgmap v40: 0 pgs: ; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:33.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:34.377 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52028]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-24T16:41:34.377 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52028]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.377 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52028]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.377 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52028]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.377 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-24T16:41:34.378 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.378 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52032]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.378 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52032]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: osdmap e18: 4 total, 3 up, 4 in 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.378 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.379 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52036]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme1n1 2026-02-24T16:41:34.379 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52036]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.379 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52036]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.379 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52036]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.484 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-24T16:41:34.484 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.484 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52580]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.484 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52580]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.485 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-24T16:41:34.485 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.485 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52584]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.485 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52584]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: osdmap e18: 4 total, 3 up, 4 in 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.486 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme1n1 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52588]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52588]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52592]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme1n1 2026-02-24T16:41:34.487 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52592]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.488 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52592]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.488 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:34 trial183 sudo[52592]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:34.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: osdmap e18: 4 total, 3 up, 4 in 2026-02-24T16:41:34.489 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66054]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme1n1 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66054]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.490 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66054]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.491 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66054]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.491 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66058]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme1n1 2026-02-24T16:41:34.491 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66058]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.491 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66058]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.491 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66058]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66046]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66046]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66046]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66046]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66050]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-24T16:41:34.492 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.493 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66050]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.493 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:41:34 trial151 sudo[66050]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:34.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52040]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme1n1 2026-02-24T16:41:34.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52040]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-24T16:41:34.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52040]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-24T16:41:34.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:34 trial153 sudo[52040]: pam_unix(sudo:session): session closed for user root 2026-02-24T16:41:35.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: osdmap e19: 4 total, 3 up, 4 in 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:35 trial183 ceph-mon[43553]: pgmap v43: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: osdmap e19: 4 total, 3 up, 4 in 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:35.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.487 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:35 trial151 ceph-mon[39453]: pgmap v43: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: osdmap e19: 4 total, 3 up, 4 in 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:35.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:35 trial153 ceph-mon[44017]: pgmap v43: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:36.470 INFO:teuthology.orchestra.run.trial191.stdout:Created osd(s) 3 on host 'trial191' 2026-02-24T16:41:36.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:36 trial183 ceph-mon[43553]: osdmap e20: 4 total, 3 up, 4 in 2026-02-24T16:41:36.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:36 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:36.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:36 trial183 ceph-mon[43553]: mgrmap e14: x(active, since 68s) 2026-02-24T16:41:36.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:36 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:36 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:36 trial151 ceph-mon[39453]: osdmap e20: 4 total, 3 up, 4 in 2026-02-24T16:41:36.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:36.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:36 trial151 ceph-mon[39453]: mgrmap e14: x(active, since 68s) 2026-02-24T16:41:36.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:36 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.507 DEBUG:teuthology.orchestra.run.trial191:osd.3> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.3.service 2026-02-24T16:41:36.510 INFO:tasks.cephadm:Waiting for 4 OSDs to come up... 2026-02-24T16:41:36.511 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd stat -f json 2026-02-24T16:41:36.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:36 trial153 ceph-mon[44017]: osdmap e20: 4 total, 3 up, 4 in 2026-02-24T16:41:36.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:36 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:36.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:36 trial153 ceph-mon[44017]: mgrmap e14: x(active, since 68s) 2026-02-24T16:41:36.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:36 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:36 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:36.630 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:36.841 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:41:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:41:36.581+0000 7ff9d3a50780 -1 osd.3 0 log_to_monitors true 2026-02-24T16:41:36.846 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:36.885 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":20,"num_osds":4,"num_up_osds":3,"osd_up_since":1771951290,"num_in_osds":4,"osd_in_since":1771951289,"num_remapped_pgs":0} 2026-02-24T16:41:37.613 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: pgmap v45: 1 pgs: 1 unknown; 0 B data, 82 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2748611999' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.614 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.615 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:37 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: pgmap v45: 1 pgs: 1 unknown; 0 B data, 82 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2748611999' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:37.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:37 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: pgmap v45: 1 pgs: 1 unknown; 0 B data, 82 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2748611999' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:37.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:37 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:37.886 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd stat -f json 2026-02-24T16:41:38.004 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:38.222 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:38.261 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1771951290,"num_in_osds":4,"osd_in_since":1771951289,"num_remapped_pgs":0} 2026-02-24T16:41:38.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.485 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: osdmap e21: 4 total, 3 up, 4 in 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3894559265' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:38.486 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:38 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:38.610 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:41:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:41:38.484+0000 7ff9cf9d2640 -1 osd.3 0 waiting for initial osdmap 2026-02-24T16:41:38.611 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:41:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:41:38.487+0000 7ff9c7891640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T16:41:38.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: osdmap e21: 4 total, 3 up, 4 in 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3894559265' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:38.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:38.735 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:38 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:38.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: osdmap e21: 4 total, 3 up, 4 in 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3894559265' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:38.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:38.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:38.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:38.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-24T16:41:38.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:38 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:39.262 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd stat -f json 2026-02-24T16:41:39.381 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:39.593 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: Detected new or changed devices on trial191 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: Adjusting osd_memory_target on trial191 to 89167M 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]}]': finished 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: osdmap e22: 4 total, 3 up, 4 in 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.593 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:39 trial151 ceph-mon[39453]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:39.628 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":23,"num_osds":4,"num_up_osds":4,"osd_up_since":1771951299,"num_in_osds":4,"osd_in_since":1771951289,"num_remapped_pgs":1} 2026-02-24T16:41:39.628 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd dump --format=json 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: Detected new or changed devices on trial191 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: Adjusting osd_memory_target on trial191 to 89167M 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]}]': finished 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: osdmap e22: 4 total, 3 up, 4 in 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:39 trial183 ceph-mon[43553]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:39.745 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: Detected new or changed devices on trial191 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: Adjusting osd_memory_target on trial191 to 89167M 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial191", "root=default"]}]': finished 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: osdmap e22: 4 total, 3 up, 4 in 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:39.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:39 trial153 ceph-mon[44017]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-24T16:41:39.951 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:39.951 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":23,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","created":"2026-02-24T16:40:06.321692+0000","modified":"2026-02-24T16:41:39.474196+0000","last_up_change":"2026-02-24T16:41:39.474196+0000","last_in_change":"2026-02-24T16:41:29.857190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:41:32.754795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40ba569b-fd87-4dcc-8ca5-fa963049100c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6802","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6803","nonce":110062519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6804","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6805","nonce":110062519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6808","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6809","nonce":110062519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6806","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6807","nonce":110062519}]},"public_addr":"10.20.193.151:6803/110062519","cluster_addr":"10.20.193.151:6805/110062519","heartbeat_back_addr":"10.20.193.151:6809/110062519","heartbeat_front_addr":"10.20.193.151:6807/110062519","state":["exists","up"]},{"osd":1,"uuid":"6cb9011d-41ad-4cf7-94b0-d899df29dfc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6801","nonce":3548290599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6802","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6803","nonce":3548290599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6806","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6807","nonce":3548290599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6804","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6805","nonce":3548290599}]},"public_addr":"10.20.193.153:6801/3548290599","cluster_addr":"10.20.193.153:6803/3548290599","heartbeat_back_addr":"10.20.193.153:6807/3548290599","heartbeat_front_addr":"10.20.193.153:6805/3548290599","state":["exists","up"]},{"osd":2,"uuid":"ed27ee6a-67e6-48f8-a96b-9f87ce25870d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6800","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6801","nonce":1741249410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6803","nonce":1741249410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6807","nonce":1741249410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6805","nonce":1741249410}]},"public_addr":"10.20.193.183:6801/1741249410","cluster_addr":"10.20.193.183:6803/1741249410","heartbeat_back_addr":"10.20.193.183:6807/1741249410","heartbeat_front_addr":"10.20.193.183:6805/1741249410","state":["exists","up"]},{"osd":3,"uuid":"fef31c97-bc0f-41bd-bca9-d13596a02d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6800","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6801","nonce":3727179608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6803","nonce":3727179608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6807","nonce":3727179608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6805","nonce":3727179608}]},"public_addr":"10.20.193.191:6801/3727179608","cluster_addr":"10.20.193.191:6803/3727179608","heartbeat_back_addr":"10.20.193.191:6807/3727179608","heartbeat_front_addr":"10.20.193.191:6805/3727179608","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:14.596237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:20.899549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:29.221019+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[2,1,0]}],"primary_temp":[],"blocklist":{"10.20.193.151:0/2954129700":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6801/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/189033368":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6801/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6800/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6800/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/2703363554":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/3257725282":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/3501985314":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/17652534":"2026-02-25T16:40:16.264079+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:41:39.987 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-24T16:41:32.754795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'nonprimary_shards': '{}', 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.75, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-24T16:41:39.988 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd pool get .mgr pg_num 2026-02-24T16:41:40.104 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:40.313 INFO:teuthology.orchestra.run.trial151.stdout:pg_num: 1 2026-02-24T16:41:40.352 INFO:tasks.cephadm:Setting up client nodes... 2026-02-24T16:41:40.353 DEBUG:teuthology.orchestra.run.trial051:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:40.469 INFO:teuthology.orchestra.run.trial051.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:40.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: purged_snaps scrub starts 2026-02-24T16:41:40.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: purged_snaps scrub ok 2026-02-24T16:41:40.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608] boot 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: osdmap e23: 4 total, 4 up, 4 in 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/236296354' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/145624291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3587479299' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: purged_snaps scrub starts 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: purged_snaps scrub ok 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608] boot 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: osdmap e23: 4 total, 4 up, 4 in 2026-02-24T16:41:40.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/236296354' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:40.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/145624291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:40.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3587479299' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2026-02-24T16:41:40.741 INFO:teuthology.orchestra.run.trial051.stdout:[client.0] 2026-02-24T16:41:40.741 INFO:teuthology.orchestra.run.trial051.stdout: key = AQDE1J1pWDg8LBAAWQm7/Y/KX3HRq55Q9nqmXg== 2026-02-24T16:41:40.798 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:41:40.798 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2026-02-24T16:41:40.798 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T16:41:40.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: purged_snaps scrub starts 2026-02-24T16:41:40.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: purged_snaps scrub ok 2026-02-24T16:41:40.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: osd.3 [v2:10.20.193.191:6800/3727179608,v1:10.20.193.191:6801/3727179608] boot 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: osdmap e23: 4 total, 4 up, 4 in 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/236296354' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/145624291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:40.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3587479299' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2026-02-24T16:41:40.865 DEBUG:teuthology.orchestra.run.trial052:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:40.983 INFO:teuthology.orchestra.run.trial052.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:41.253 INFO:teuthology.orchestra.run.trial052.stdout:[client.1] 2026-02-24T16:41:41.253 INFO:teuthology.orchestra.run.trial052.stdout: key = AQDF1J1pDL4dDxAAb0xiPgLHPxX1VDWeHX3hmA== 2026-02-24T16:41:41.310 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:41:41.310 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2026-02-24T16:41:41.311 DEBUG:teuthology.orchestra.run.trial052:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-24T16:41:41.389 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:41.507 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: osdmap e24: 4 total, 4 up, 4 in 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: osdmap e24: 4 total, 4 up, 4 in 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.766 INFO:teuthology.orchestra.run.trial151.stdout:[client.2] 2026-02-24T16:41:41.766 INFO:teuthology.orchestra.run.trial151.stdout: key = AQDF1J1pV9K9LRAAobUv4B6/Ifr6SvI0fh/I+w== 2026-02-24T16:41:41.803 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:41:41.803 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2026-02-24T16:41:41.803 DEBUG:teuthology.orchestra.run.trial151:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: osdmap e24: 4 total, 4 up, 4 in 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.51:0/3217834331' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:41.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.52:0/1087417412' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:41.838 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.3 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:41.957 INFO:teuthology.orchestra.run.trial153.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.b/config 2026-02-24T16:41:42.217 INFO:teuthology.orchestra.run.trial153.stdout:[client.3] 2026-02-24T16:41:42.217 INFO:teuthology.orchestra.run.trial153.stdout: key = AQDG1J1pdhToDBAAVNYe+8Fbfuj05XWZx5V1bQ== 2026-02-24T16:41:42.277 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:41:42.277 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/ceph/ceph.client.3.keyring 2026-02-24T16:41:42.277 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2026-02-24T16:41:42.312 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.4 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:42.430 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:41:42.691 INFO:teuthology.orchestra.run.trial183.stdout:[client.4] 2026-02-24T16:41:42.691 INFO:teuthology.orchestra.run.trial183.stdout: key = AQDG1J1p6qoqKRAAxlaxYQE7HEKNa6u1lWwF/A== 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: osdmap e25: 4 total, 4 up, 4 in 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: osdmap e26: 4 total, 4 up, 4 in 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.692 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:42 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2088704730' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.729 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:41:42.729 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/ceph.client.4.keyring 2026-02-24T16:41:42.729 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 0644 /etc/ceph/ceph.client.4.keyring 2026-02-24T16:41:42.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: osdmap e25: 4 total, 4 up, 4 in 2026-02-24T16:41:42.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: osdmap e26: 4 total, 4 up, 4 in 2026-02-24T16:41:42.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:42 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2088704730' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.764 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph auth get-or-create client.5 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-24T16:41:42.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: osdmap e25: 4 total, 4 up, 4 in 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1960569709' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: osdmap e26: 4 total, 4 up, 4 in 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:42.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:42 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2088704730' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:42.883 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:43.144 INFO:teuthology.orchestra.run.trial191.stdout:[client.5] 2026-02-24T16:41:43.144 INFO:teuthology.orchestra.run.trial191.stdout: key = AQDH1J1p61ukCBAAwB7QibTaz9ENdJ+nEJb54Q== 2026-02-24T16:41:43.185 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:41:43.185 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/ceph/ceph.client.5.keyring 2026-02-24T16:41:43.185 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0644 /etc/ceph/ceph.client.5.keyring 2026-02-24T16:41:43.221 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph config log 1 --format=json 2026-02-24T16:41:43.337 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:43.550 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:43.550 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: pgmap v54: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:43.550 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.550 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.551 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/360490531' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.551 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.551 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.551 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:43 trial151 ceph-mon[39453]: osdmap e27: 4 total, 4 up, 4 in 2026-02-24T16:41:43.589 INFO:teuthology.orchestra.run.trial151.stdout:[{"version":13,"timestamp":"2026-02-24T16:41:38.404851+0000","name":"","changes":[{"name":"osd/host:trial191/osd_memory_target","new_value":"93498626457"}]}] 2026-02-24T16:41:43.590 INFO:tasks.ceph_manager:config epoch is 13 2026-02-24T16:41:43.591 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T16:41:43.591 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2026-02-24T16:41:43.591 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph mgr dump --format=json 2026-02-24T16:41:43.705 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:43.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: pgmap v54: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:43.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/360490531' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:43 trial153 ceph-mon[44017]: osdmap e27: 4 total, 4 up, 4 in 2026-02-24T16:41:43.946 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:43.981 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":14,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6800","nonce":3174062078},{"type":"v1","addr":"10.20.193.151:6801","nonce":3174062078}]},"active_addr":"10.20.193.151:6801/3174062078","active_change":"2026-02-24T16:40:26.539172+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://10.20.193.151:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.151:0","nonce":208958308}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.151:0","nonce":1437396385}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.151:0","nonce":3182543798}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.151:0","nonce":3879886146}]}]} 2026-02-24T16:41:43.985 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2026-02-24T16:41:43.986 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2026-02-24T16:41:43.986 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd dump --format=json 2026-02-24T16:41:43.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: pgmap v54: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/360490531' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2467798156' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.5", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-24T16:41:43.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:43 trial183 ceph-mon[43553]: osdmap e27: 4 total, 4 up, 4 in 2026-02-24T16:41:44.099 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.312 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:44.313 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":27,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","created":"2026-02-24T16:40:06.321692+0000","modified":"2026-02-24T16:41:43.180619+0000","last_up_change":"2026-02-24T16:41:39.474196+0000","last_in_change":"2026-02-24T16:41:29.857190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:41:32.754795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40ba569b-fd87-4dcc-8ca5-fa963049100c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6802","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6803","nonce":110062519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6804","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6805","nonce":110062519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6808","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6809","nonce":110062519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6806","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6807","nonce":110062519}]},"public_addr":"10.20.193.151:6803/110062519","cluster_addr":"10.20.193.151:6805/110062519","heartbeat_back_addr":"10.20.193.151:6809/110062519","heartbeat_front_addr":"10.20.193.151:6807/110062519","state":["exists","up"]},{"osd":1,"uuid":"6cb9011d-41ad-4cf7-94b0-d899df29dfc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6801","nonce":3548290599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6802","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6803","nonce":3548290599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6806","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6807","nonce":3548290599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6804","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6805","nonce":3548290599}]},"public_addr":"10.20.193.153:6801/3548290599","cluster_addr":"10.20.193.153:6803/3548290599","heartbeat_back_addr":"10.20.193.153:6807/3548290599","heartbeat_front_addr":"10.20.193.153:6805/3548290599","state":["exists","up"]},{"osd":2,"uuid":"ed27ee6a-67e6-48f8-a96b-9f87ce25870d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6800","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6801","nonce":1741249410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6803","nonce":1741249410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6807","nonce":1741249410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6805","nonce":1741249410}]},"public_addr":"10.20.193.183:6801/1741249410","cluster_addr":"10.20.193.183:6803/1741249410","heartbeat_back_addr":"10.20.193.183:6807/1741249410","heartbeat_front_addr":"10.20.193.183:6805/1741249410","state":["exists","up"]},{"osd":3,"uuid":"fef31c97-bc0f-41bd-bca9-d13596a02d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6800","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6801","nonce":3727179608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6803","nonce":3727179608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6807","nonce":3727179608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6805","nonce":3727179608}]},"public_addr":"10.20.193.191:6801/3727179608","cluster_addr":"10.20.193.191:6803/3727179608","heartbeat_back_addr":"10.20.193.191:6807/3727179608","heartbeat_front_addr":"10.20.193.191:6805/3727179608","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:14.596237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:20.899549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:29.221019+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:37.574805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.151:0/2954129700":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6801/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/189033368":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6801/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6800/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6800/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/2703363554":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/3257725282":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/3501985314":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/17652534":"2026-02-25T16:40:16.264079+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:41:44.349 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2026-02-24T16:41:44.350 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd dump --format=json 2026-02-24T16:41:44.467 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.675 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:44.675 INFO:teuthology.orchestra.run.trial151.stdout:{"epoch":27,"fsid":"6deba16a-119f-11f1-a847-d404e6e7d460","created":"2026-02-24T16:40:06.321692+0000","modified":"2026-02-24T16:41:43.180619+0000","last_up_change":"2026-02-24T16:41:39.474196+0000","last_in_change":"2026-02-24T16:41:29.857190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:41:32.754795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"40ba569b-fd87-4dcc-8ca5-fa963049100c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6802","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6803","nonce":110062519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6804","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6805","nonce":110062519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6808","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6809","nonce":110062519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.151:6806","nonce":110062519},{"type":"v1","addr":"10.20.193.151:6807","nonce":110062519}]},"public_addr":"10.20.193.151:6803/110062519","cluster_addr":"10.20.193.151:6805/110062519","heartbeat_back_addr":"10.20.193.151:6809/110062519","heartbeat_front_addr":"10.20.193.151:6807/110062519","state":["exists","up"]},{"osd":1,"uuid":"6cb9011d-41ad-4cf7-94b0-d899df29dfc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6801","nonce":3548290599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6802","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6803","nonce":3548290599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6806","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6807","nonce":3548290599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6804","nonce":3548290599},{"type":"v1","addr":"10.20.193.153:6805","nonce":3548290599}]},"public_addr":"10.20.193.153:6801/3548290599","cluster_addr":"10.20.193.153:6803/3548290599","heartbeat_back_addr":"10.20.193.153:6807/3548290599","heartbeat_front_addr":"10.20.193.153:6805/3548290599","state":["exists","up"]},{"osd":2,"uuid":"ed27ee6a-67e6-48f8-a96b-9f87ce25870d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6800","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6801","nonce":1741249410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6803","nonce":1741249410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6807","nonce":1741249410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":1741249410},{"type":"v1","addr":"10.20.193.183:6805","nonce":1741249410}]},"public_addr":"10.20.193.183:6801/1741249410","cluster_addr":"10.20.193.183:6803/1741249410","heartbeat_back_addr":"10.20.193.183:6807/1741249410","heartbeat_front_addr":"10.20.193.183:6805/1741249410","state":["exists","up"]},{"osd":3,"uuid":"fef31c97-bc0f-41bd-bca9-d13596a02d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6800","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6801","nonce":3727179608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6803","nonce":3727179608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6807","nonce":3727179608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3727179608},{"type":"v1","addr":"10.20.193.191:6805","nonce":3727179608}]},"public_addr":"10.20.193.191:6801/3727179608","cluster_addr":"10.20.193.191:6803/3727179608","heartbeat_back_addr":"10.20.193.191:6807/3727179608","heartbeat_front_addr":"10.20.193.191:6805/3727179608","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:14.596237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:20.899549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:29.221019+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T16:41:37.574805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.151:0/2954129700":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6801/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/189033368":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6801/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:6800/241515977":"2026-02-25T16:40:26.539061+0000","10.20.193.151:6800/3427686791":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/2703363554":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/3257725282":"2026-02-25T16:40:16.264079+0000","10.20.193.151:0/3501985314":"2026-02-25T16:40:26.539061+0000","10.20.193.151:0/17652534":"2026-02-25T16:40:16.264079+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:41:44.676 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/605183222' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2026-02-24T16:41:44.676 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1851949404' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2026-02-24T16:41:44.676 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1028341068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:44.712 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph tell osd.0 flush_pg_stats 2026-02-24T16:41:44.713 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph tell osd.1 flush_pg_stats 2026-02-24T16:41:44.713 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph tell osd.2 flush_pg_stats 2026-02-24T16:41:44.714 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph tell osd.3 flush_pg_stats 2026-02-24T16:41:44.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/605183222' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2026-02-24T16:41:44.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1851949404' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2026-02-24T16:41:44.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1028341068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:44.839 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.840 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.840 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.842 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:44.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/605183222' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2026-02-24T16:41:44.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1851949404' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2026-02-24T16:41:44.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1028341068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:45.013 INFO:teuthology.orchestra.run.trial151.stdout:98784247812 2026-02-24T16:41:45.013 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.3 2026-02-24T16:41:45.014 INFO:teuthology.orchestra.run.trial151.stdout:68719476741 2026-02-24T16:41:45.015 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.2 2026-02-24T16:41:45.016 INFO:teuthology.orchestra.run.trial151.stdout:34359738376 2026-02-24T16:41:45.017 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.0 2026-02-24T16:41:45.021 INFO:teuthology.orchestra.run.trial151.stdout:51539607559 2026-02-24T16:41:45.021 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.1 2026-02-24T16:41:45.136 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:45.177 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:45.178 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:45.180 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:45.364 INFO:teuthology.orchestra.run.trial151.stdout:98784247811 2026-02-24T16:41:45.391 INFO:teuthology.orchestra.run.trial151.stdout:68719476740 2026-02-24T16:41:45.399 INFO:teuthology.orchestra.run.trial151.stdout:51539607558 2026-02-24T16:41:45.401 INFO:teuthology.orchestra.run.trial151.stdout:34359738375 2026-02-24T16:41:45.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.3 2026-02-24T16:41:45.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.2 2026-02-24T16:41:45.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607559 got 51539607558 for osd.1 2026-02-24T16:41:45.449 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738376 got 34359738375 for osd.0 2026-02-24T16:41:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: pgmap v56: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/111904747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2094386378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3691116210' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/535364674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2729718065' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: pgmap v56: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/111904747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2094386378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3691116210' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/535364674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2729718065' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: pgmap v56: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/111904747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2094386378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3691116210' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/535364674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2729718065' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:46.405 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.3 2026-02-24T16:41:46.440 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.2 2026-02-24T16:41:46.443 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.1 2026-02-24T16:41:46.450 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd last-stat-seq osd.0 2026-02-24T16:41:46.521 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:46.566 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:46.567 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:46.610 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:46.746 INFO:teuthology.orchestra.run.trial151.stdout:98784247813 2026-02-24T16:41:46.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.3 2026-02-24T16:41:46.784 DEBUG:teuthology.parallel:result is None 2026-02-24T16:41:46.794 INFO:teuthology.orchestra.run.trial151.stdout:51539607559 2026-02-24T16:41:46.801 INFO:teuthology.orchestra.run.trial151.stdout:68719476742 2026-02-24T16:41:46.834 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607559 got 51539607559 for osd.1 2026-02-24T16:41:46.834 DEBUG:teuthology.parallel:result is None 2026-02-24T16:41:46.839 INFO:teuthology.orchestra.run.trial151.stdout:34359738377 2026-02-24T16:41:46.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476741 got 68719476742 for osd.2 2026-02-24T16:41:46.843 DEBUG:teuthology.parallel:result is None 2026-02-24T16:41:46.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738376 got 34359738377 for osd.0 2026-02-24T16:41:46.877 DEBUG:teuthology.parallel:result is None 2026-02-24T16:41:46.877 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2026-02-24T16:41:46.877 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph pg dump --format=json 2026-02-24T16:41:47.031 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:47.237 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:47.237 INFO:teuthology.orchestra.run.trial151.stderr:dumped all 2026-02-24T16:41:47.273 INFO:teuthology.orchestra.run.trial151.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2026-02-24T16:41:46.549463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110492,"kb_used_data":2636,"kb_used_omap":36,"kb_used_meta":107739,"kb_avail":2935902308,"statfs":{"total":3006477107200,"available":3006363963392,"internally_reserved":0,"allocated":2699264,"data_stored":2107416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37074,"internal_metadata":110325550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000645"},"pg_stats":[{"pgid":"1.0","version":"19'152","reported_seq":209,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-24T16:41:43.192537+0000","last_change":"2026-02-24T16:41:43.192377+0000","last_active":"2026-02-24T16:41:43.192537+0000","last_peered":"2026-02-24T16:41:43.192537+0000","last_clean":"2026-02-24T16:41:43.192537+0000","last_became_active":"2026-02-24T16:41:43.192210+0000","last_became_peered":"2026-02-24T16:41:43.192210+0000","last_unstale":"2026-02-24T16:41:43.192537+0000","last_undegraded":"2026-02-24T16:41:43.192537+0000","last_fullsized":"2026-02-24T16:41:43.192537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:41:33.115964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:41:33.115964+0000","last_clean_scrub_stamp":"2026-02-24T16:41:33.115964+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:40:51.371687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6824,"internal_metadata":27583832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":11,"kb_used_meta":26932,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11375,"internal_metadata":27579281},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7485,"internal_metadata":27583171},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27188,"kb_used_data":224,"kb_used_omap":11,"kb_used_meta":26932,"kb_avail":733976012,"statfs":{"total":751619276800,"available":751591436288,"internally_reserved":0,"allocated":229376,"data_stored":84078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11390,"internal_metadata":27579266},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:41:47.273 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph pg dump --format=json 2026-02-24T16:41:47.386 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:47.591 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:47.591 INFO:teuthology.orchestra.run.trial151.stderr:dumped all 2026-02-24T16:41:47.628 INFO:teuthology.orchestra.run.trial151.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2026-02-24T16:41:46.549463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110492,"kb_used_data":2636,"kb_used_omap":36,"kb_used_meta":107739,"kb_avail":2935902308,"statfs":{"total":3006477107200,"available":3006363963392,"internally_reserved":0,"allocated":2699264,"data_stored":2107416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37074,"internal_metadata":110325550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000645"},"pg_stats":[{"pgid":"1.0","version":"19'152","reported_seq":209,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-24T16:41:43.192537+0000","last_change":"2026-02-24T16:41:43.192377+0000","last_active":"2026-02-24T16:41:43.192537+0000","last_peered":"2026-02-24T16:41:43.192537+0000","last_clean":"2026-02-24T16:41:43.192537+0000","last_became_active":"2026-02-24T16:41:43.192210+0000","last_became_peered":"2026-02-24T16:41:43.192210+0000","last_unstale":"2026-02-24T16:41:43.192537+0000","last_undegraded":"2026-02-24T16:41:43.192537+0000","last_fullsized":"2026-02-24T16:41:43.192537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:41:33.115964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:41:33.115964+0000","last_clean_scrub_stamp":"2026-02-24T16:41:33.115964+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:40:51.371687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":189,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6824,"internal_metadata":27583832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":11,"kb_used_meta":26932,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11375,"internal_metadata":27579281},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":674446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7485,"internal_metadata":27583171},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27188,"kb_used_data":224,"kb_used_omap":11,"kb_used_meta":26932,"kb_avail":733976012,"statfs":{"total":751619276800,"available":751591436288,"internally_reserved":0,"allocated":229376,"data_stored":84078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11390,"internal_metadata":27579266},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:41:47.629 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2026-02-24T16:41:47.630 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T16:41:47.630 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2026-02-24T16:41:47.630 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph health --format=json 2026-02-24T16:41:47.746 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:47.767 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:47 trial151 ceph-mon[39453]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:47.768 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2551413907' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:47.768 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3345367934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:47.768 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2586594471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:47.768 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1785424876' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:47 trial183 ceph-mon[43553]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2551413907' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3345367934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2586594471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1785424876' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:47.984 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:47.984 INFO:teuthology.orchestra.run.trial151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T16:41:48.021 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2026-02-24T16:41:48.021 INFO:tasks.cephadm:Setup complete, yielding 2026-02-24T16:41:48.021 INFO:teuthology.run_tasks:Running task cephadm.shell... 2026-02-24T16:41:48.034 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@trial151.front.sepia.ceph.com 2026-02-24T16:41:48.034 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- bash -c 'ceph orch status' 2026-02-24T16:41:48.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:47 trial153 ceph-mon[44017]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:48.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2551413907' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-24T16:41:48.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3345367934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-24T16:41:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2586594471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-24T16:41:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1785424876' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-24T16:41:48.151 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:48.368 INFO:teuthology.orchestra.run.trial151.stdout:Backend: cephadm 2026-02-24T16:41:48.368 INFO:teuthology.orchestra.run.trial151.stdout:Available: Yes 2026-02-24T16:41:48.368 INFO:teuthology.orchestra.run.trial151.stdout:Paused: No 2026-02-24T16:41:48.404 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- bash -c 'ceph orch ps' 2026-02-24T16:41:48.519 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:48.633 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:48 trial151 ceph-mon[39453]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:48.633 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:48 trial151 ceph-mon[39453]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:48.633 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2133148070' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:mgr.x trial151 *:9283,8765 running (101s) 31s ago 101s 548M - 20.3.0-5507-g618c318a c2b2dcd214a8 1eb53b3e2fe3 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:mon.a trial151 running (101s) 31s ago 102s 43.5M 2048M 20.3.0-5507-g618c318a c2b2dcd214a8 448ac3ea15d1 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:mon.b trial153 running (55s) 27s ago 55s 37.8M 2048M 20.3.0-5507-g618c318a c2b2dcd214a8 14e923b0d61e 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:mon.c trial183 running (62s) 11s ago 61s 39.7M 2048M 20.3.0-5507-g618c318a c2b2dcd214a8 9a228b3e7f16 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:osd.0 trial151 running (35s) 31s ago 37s 30.4M 82.0G 20.3.0-5507-g618c318a c2b2dcd214a8 23fbc7cc9f71 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:osd.1 trial153 running (29s) 27s ago 29s 29.0M 86.0G 20.3.0-5507-g618c318a c2b2dcd214a8 dc0ca4deef67 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:osd.2 trial183 running (21s) 11s ago 23s 41.5M 86.0G 20.3.0-5507-g618c318a c2b2dcd214a8 a2d875db87ff 2026-02-24T16:41:48.741 INFO:teuthology.orchestra.run.trial151.stdout:osd.3 trial191 running (12s) 11s ago 14s 27.0M 87.0G 20.3.0-5507-g618c318a c2b2dcd214a8 857ad55401df 2026-02-24T16:41:48.778 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- bash -c 'ceph orch host ls' 2026-02-24T16:41:48.891 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:48.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:48 trial183 ceph-mon[43553]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:48.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:48 trial183 ceph-mon[43553]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:48.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2133148070' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2026-02-24T16:41:49.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:48 trial153 ceph-mon[44017]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:49.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:48 trial153 ceph-mon[44017]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:49.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2133148070' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:HOST ADDR LABELS STATUS 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial051 10.20.193.51 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial052 10.20.193.52 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial151 10.20.193.151 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial153 10.20.193.153 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial183 10.20.193.183 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:trial191 10.20.193.191 2026-02-24T16:41:49.104 INFO:teuthology.orchestra.run.trial151.stdout:6 hosts in cluster 2026-02-24T16:41:49.140 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- bash -c 'ceph orch device ls' 2026-02-24T16:41:49.255 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial051 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B47B0F 1788G No 74s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial051 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B47B13 1788G No 74s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial052 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B47DC4 1788G No 71s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial052 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B47A7F 1788G No 71s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial151 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B00266 1788G No 31s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.472 INFO:teuthology.orchestra.run.trial151.stdout:trial151 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B002D5 1788G No 31s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial153 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_092510AB6933 1788G No 26s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial153 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_092510AB6993 1788G No 26s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial183 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B003FB 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial183 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B0040D 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial191 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_092510AB6B6F 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.473 INFO:teuthology.orchestra.run.trial151.stdout:trial191 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_092510AB6A5A 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-24T16:41:49.510 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- bash -c 'ceph osd lspools' 2026-02-24T16:41:49.625 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:49.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:49 trial151 ceph-mon[39453]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:49.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:49 trial151 ceph-mon[39453]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:49.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:49 trial151 ceph-mon[39453]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:49.830 INFO:teuthology.orchestra.run.trial151.stdout:1 .mgr 2026-02-24T16:41:49.868 INFO:teuthology.run_tasks:Running task nvmeof... 2026-02-24T16:41:49.896 INFO:tasks.nvmeof:[nvmeof]: deploying nvmeof gateway... 2026-02-24T16:41:49.896 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.a on trial151 2026-02-24T16:41:49.896 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.b on trial153 2026-02-24T16:41:49.896 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.c on trial183 2026-02-24T16:41:49.896 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.d on trial191 2026-02-24T16:41:49.897 INFO:tasks.nvmeof:[nvmeof]: ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:devel 2026-02-24T16:41:49.897 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:devel 2026-02-24T16:41:49.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:49 trial183 ceph-mon[43553]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:49.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:49 trial183 ceph-mon[43553]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:49.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:49 trial183 ceph-mon[43553]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.011 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:50.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:49 trial153 ceph-mon[44017]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:49 trial153 ceph-mon[44017]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:49 trial153 ceph-mon[44017]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.262 INFO:tasks.nvmeof:[nvmeof]: ceph osd pool create mypool 2026-02-24T16:41:50.262 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph osd pool create mypool 2026-02-24T16:41:50.378 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:50.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1973886677' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2659826931' entity='client.admin' 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1973886677' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2659826931' entity='client.admin' 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2026-02-24T16:41:51.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:51.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:51.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1973886677' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2026-02-24T16:41:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2659826931' entity='client.admin' 2026-02-24T16:41:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2026-02-24T16:41:51.539 INFO:teuthology.orchestra.run.trial151.stderr:pool 'mypool' created 2026-02-24T16:41:51.581 INFO:tasks.nvmeof:[nvmeof]: rbd pool init mypool 2026-02-24T16:41:51.581 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- rbd pool init mypool 2026-02-24T16:41:51.696 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:51.810 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:51 trial151 ceph-mon[39453]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:51.810 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2026-02-24T16:41:51.811 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:51 trial151 ceph-mon[39453]: osdmap e28: 4 total, 4 up, 4 in 2026-02-24T16:41:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:51 trial183 ceph-mon[43553]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2026-02-24T16:41:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:51 trial183 ceph-mon[43553]: osdmap e28: 4 total, 4 up, 4 in 2026-02-24T16:41:52.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:51 trial153 ceph-mon[44017]: pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:52.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3244232149' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2026-02-24T16:41:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:51 trial153 ceph-mon[44017]: osdmap e28: 4 total, 4 up, 4 in 2026-02-24T16:41:52.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2026-02-24T16:41:52.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:52 trial183 ceph-mon[43553]: osdmap e29: 4 total, 4 up, 4 in 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2026-02-24T16:41:52.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2026-02-24T16:41:52.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:52 trial151 ceph-mon[39453]: osdmap e29: 4 total, 4 up, 4 in 2026-02-24T16:41:53.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:53.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:53.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:53.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2026-02-24T16:41:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3101629753' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2026-02-24T16:41:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:52 trial153 ceph-mon[44017]: osdmap e29: 4 total, 4 up, 4 in 2026-02-24T16:41:53.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:53 trial183 ceph-mon[43553]: pgmap v62: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:53.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:53 trial183 ceph-mon[43553]: osdmap e30: 4 total, 4 up, 4 in 2026-02-24T16:41:53.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:53 trial151 ceph-mon[39453]: pgmap v62: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:53.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:53 trial151 ceph-mon[39453]: osdmap e30: 4 total, 4 up, 4 in 2026-02-24T16:41:54.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:53 trial153 ceph-mon[44017]: pgmap v62: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:54.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:53 trial153 ceph-mon[44017]: osdmap e30: 4 total, 4 up, 4 in 2026-02-24T16:41:54.578 INFO:tasks.nvmeof:[nvmeof]: ceph orch apply nvmeof mypool mygroup0 2026-02-24T16:41:54.579 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch apply nvmeof mypool mygroup0 --placement 'trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d' 2026-02-24T16:41:54.694 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:54.920 INFO:teuthology.orchestra.run.trial151.stdout:Scheduled nvmeof.mypool.mygroup0 update... 2026-02-24T16:41:54.959 INFO:tasks.nvmeof:[nvmeof]: creating 60 images 2026-02-24T16:41:54.959 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- rbd create mypool/myimage1 --size 8192 ; rbd create mypool/myimage2 --size 8192 ; rbd create mypool/myimage3 --size 8192 ; rbd create mypool/myimage4 --size 8192 ; rbd create mypool/myimage5 --size 8192 ; rbd create mypool/myimage6 --size 8192 ; rbd create mypool/myimage7 --size 8192 ; rbd create mypool/myimage8 --size 8192 ; rbd create mypool/myimage9 --size 8192 ; rbd create mypool/myimage10 --size 8192 ; rbd create mypool/myimage11 --size 8192 ; rbd create mypool/myimage12 --size 8192 ; rbd create mypool/myimage13 --size 8192 ; rbd create mypool/myimage14 --size 8192 ; rbd create mypool/myimage15 --size 8192 ; rbd create mypool/myimage16 --size 8192 ; rbd create mypool/myimage17 --size 8192 ; rbd create mypool/myimage18 --size 8192 ; rbd create mypool/myimage19 --size 8192 ; rbd create mypool/myimage20 --size 8192 ; rbd create mypool/myimage21 --size 8192 ; rbd create mypool/myimage22 --size 8192 ; rbd create mypool/myimage23 --size 8192 ; rbd create mypool/myimage24 --size 8192 ; rbd create mypool/myimage25 --size 8192 ; rbd create mypool/myimage26 --size 8192 ; rbd create mypool/myimage27 --size 8192 ; rbd create mypool/myimage28 --size 8192 ; rbd create mypool/myimage29 --size 8192 ; rbd create mypool/myimage30 --size 8192 ; rbd create mypool/myimage31 --size 8192 ; rbd create mypool/myimage32 --size 8192 ; rbd create mypool/myimage33 --size 8192 ; rbd create mypool/myimage34 --size 8192 ; rbd create mypool/myimage35 --size 8192 ; rbd create mypool/myimage36 --size 8192 ; rbd create mypool/myimage37 --size 8192 ; rbd create mypool/myimage38 --size 8192 ; rbd create mypool/myimage39 --size 8192 ; rbd create mypool/myimage40 --size 8192 ; rbd create mypool/myimage41 --size 8192 ; rbd create mypool/myimage42 --size 8192 ; rbd create mypool/myimage43 --size 8192 ; rbd create mypool/myimage44 --size 8192 ; rbd create mypool/myimage45 --size 8192 ; rbd create mypool/myimage46 --size 8192 ; rbd create mypool/myimage47 --size 8192 ; rbd create mypool/myimage48 --size 8192 ; rbd create mypool/myimage49 --size 8192 ; rbd create mypool/myimage50 --size 8192 ; rbd create mypool/myimage51 --size 8192 ; rbd create mypool/myimage52 --size 8192 ; rbd create mypool/myimage53 --size 8192 ; rbd create mypool/myimage54 --size 8192 ; rbd create mypool/myimage55 --size 8192 ; rbd create mypool/myimage56 --size 8192 ; rbd create mypool/myimage57 --size 8192 ; rbd create mypool/myimage58 --size 8192 ; rbd create mypool/myimage59 --size 8192 ; rbd create mypool/myimage60 --size 8192 ; 2026-02-24T16:41:55.074 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: osdmap e31: 4 total, 4 up, 4 in 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: pgmap v65: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:55.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: osdmap e31: 4 total, 4 up, 4 in 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: pgmap v65: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: osdmap e31: 4 total, 4 up, 4 in 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: pgmap v65: 33 pgs: 7 creating+peering, 25 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:41:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:41:56.596 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2026-02-24T16:41:56.714 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:56.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:56.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:41:56.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: Saving service nvmeof.mypool.mygroup0 spec with placement trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d 2026-02-24T16:41:56.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:41:56.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: gateway address: 10.20.193.153 from map_addr=None spec.addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: discovery address: 10.20.193.153 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:56 trial153 ceph-mon[44017]: Deploying daemon nvmeof.nvmeof.b on trial153 2026-02-24T16:41:56.828 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: Saving service nvmeof.mypool.mygroup0 spec with placement trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: gateway address: 10.20.193.153 from map_addr=None spec.addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: discovery address: 10.20.193.153 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.829 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:56 trial151 ceph-mon[39453]: Deploying daemon nvmeof.nvmeof.b on trial153 2026-02-24T16:41:56.969 DEBUG:teuthology.orchestra.run.trial151:nvmeof.nvmeof.a> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.a.service 2026-02-24T16:41:56.971 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2026-02-24T16:41:56.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:56.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:41:56.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: Saving service nvmeof.mypool.mygroup0 spec with placement trial151=nvmeof.a;trial153=nvmeof.b;trial183=nvmeof.c;trial191=nvmeof.d 2026-02-24T16:41:56.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:41:56.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: gateway address: 10.20.193.153 from map_addr=None spec.addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: discovery address: 10.20.193.153 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.153' 2026-02-24T16:41:56.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:56 trial183 ceph-mon[43553]: Deploying daemon nvmeof.nvmeof.b on trial153 2026-02-24T16:41:57.092 INFO:teuthology.orchestra.run.trial153.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.b/config 2026-02-24T16:41:57.356 DEBUG:teuthology.orchestra.run.trial153:nvmeof.nvmeof.b> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.b.service 2026-02-24T16:41:57.358 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2026-02-24T16:41:57.474 INFO:teuthology.orchestra.run.trial183.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.c/config 2026-02-24T16:41:57.689 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:57 trial183 ceph-mon[43553]: pgmap v66: 33 pgs: 26 active+clean, 7 creating+peering; 589 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 51 op/s 2026-02-24T16:41:57.727 DEBUG:teuthology.orchestra.run.trial183:nvmeof.nvmeof.c> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.c.service 2026-02-24T16:41:57.730 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2026-02-24T16:41:57.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:57 trial153 ceph-mon[44017]: pgmap v66: 33 pgs: 26 active+clean, 7 creating+peering; 589 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 51 op/s 2026-02-24T16:41:57.850 INFO:teuthology.orchestra.run.trial191.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/config/ceph.conf 2026-02-24T16:41:57.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:57 trial151 ceph-mon[39453]: pgmap v66: 33 pgs: 26 active+clean, 7 creating+peering; 589 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 51 op/s 2026-02-24T16:41:58.106 DEBUG:teuthology.orchestra.run.trial191:nvmeof.nvmeof.d> sudo journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.d.service 2026-02-24T16:41:58.108 INFO:tasks.nvmeof:[nvmeof]: executed deploy_nvmeof successfully! 2026-02-24T16:41:58.108 INFO:tasks.nvmeof:[nvmeof]: running set_gateway_cfg... 2026-02-24T16:41:58.108 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:41:58.109 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.135 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:41:58.135 DEBUG:teuthology.orchestra.run.trial052:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.160 DEBUG:teuthology.orchestra.run.trial151:> set -ex 2026-02-24T16:41:58.161 DEBUG:teuthology.orchestra.run.trial151:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.187 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-24T16:41:58.188 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.216 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-24T16:41:58.217 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.242 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-24T16:41:58.243 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/ceph/nvmeof.env 2026-02-24T16:41:58.269 INFO:tasks.nvmeof:[nvmeof]: executed set_gateway_cfg successfully! 2026-02-24T16:41:58.269 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2026-02-24T16:41:58.282 INFO:tasks.cephadm:Waiting for ceph service nvmeof.mypool.mygroup0 to start (timeout 300)... 2026-02-24T16:41:58.282 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:41:58.397 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:58.612 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:58.613 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:41:58.650 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:41:58.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:58 trial151 ceph-mon[39453]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:58 trial151 ceph-mon[39453]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:58 trial151 ceph-mon[39453]: mgrmap e15: x(active, since 91s) 2026-02-24T16:41:58.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:58 trial151 ceph-mon[39453]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:58 trial153 ceph-mon[44017]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:58 trial153 ceph-mon[44017]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:58 trial153 ceph-mon[44017]: mgrmap e15: x(active, since 91s) 2026-02-24T16:41:58.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:58 trial153 ceph-mon[44017]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:58 trial183 ceph-mon[43553]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:58 trial183 ceph-mon[43553]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:58.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:58 trial183 ceph-mon[43553]: mgrmap e15: x(active, since 91s) 2026-02-24T16:41:58.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:58 trial183 ceph-mon[43553]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:59.652 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:41:59.766 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:41:59.880 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:59 trial151 ceph-mon[39453]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:59.880 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:59 trial151 ceph-mon[39453]: pgmap v67: 33 pgs: 33 active+clean; 596 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 36 KiB/s rd, 25 KiB/s wr, 81 op/s 2026-02-24T16:41:59.880 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:41:59 trial151 ceph-mon[39453]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:59 trial183 ceph-mon[43553]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:41:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:59 trial183 ceph-mon[43553]: pgmap v67: 33 pgs: 33 active+clean; 596 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 36 KiB/s rd, 25 KiB/s wr, 81 op/s 2026-02-24T16:41:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:41:59 trial183 ceph-mon[43553]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:41:59.984 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:41:59.984 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:00.023 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:00.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:59 trial153 ceph-mon[44017]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:42:00.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:59 trial153 ceph-mon[44017]: pgmap v67: 33 pgs: 33 active+clean; 596 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 36 KiB/s rd, 25 KiB/s wr, 81 op/s 2026-02-24T16:42:00.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:41:59 trial153 ceph-mon[44017]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:01.024 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:01.143 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:01.363 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:01.363 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:01.415 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:01 trial183 ceph-mon[43553]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:01 trial183 ceph-mon[43553]: pgmap v68: 33 pgs: 33 active+clean; 606 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 70 op/s 2026-02-24T16:42:01.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:01 trial151 ceph-mon[39453]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:01.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:01 trial151 ceph-mon[39453]: pgmap v68: 33 pgs: 33 active+clean; 606 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 70 op/s 2026-02-24T16:42:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:01 trial153 ceph-mon[44017]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:01 trial153 ceph-mon[44017]: pgmap v68: 33 pgs: 33 active+clean; 606 KiB data, 109 MiB used, 2.7 TiB / 2.7 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 70 op/s 2026-02-24T16:42:02.416 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:02.534 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:02.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:02 trial151 ceph-mon[39453]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:02.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:02 trial151 ceph-mon[39453]: osdmap e32: 4 total, 4 up, 4 in 2026-02-24T16:42:02.749 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:02.749 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:02.788 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:02.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:02 trial183 ceph-mon[43553]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:02.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:02 trial183 ceph-mon[43553]: osdmap e32: 4 total, 4 up, 4 in 2026-02-24T16:42:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:02 trial153 ceph-mon[44017]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:02 trial153 ceph-mon[44017]: osdmap e32: 4 total, 4 up, 4 in 2026-02-24T16:42:03.789 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:03.909 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:03.930 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:03 trial151 ceph-mon[39453]: pgmap v70: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 90 op/s 2026-02-24T16:42:03.931 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:03 trial151 ceph-mon[39453]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:03.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:03 trial183 ceph-mon[43553]: pgmap v70: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 90 op/s 2026-02-24T16:42:03.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:03 trial183 ceph-mon[43553]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:03 trial153 ceph-mon[44017]: pgmap v70: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 90 op/s 2026-02-24T16:42:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:03 trial153 ceph-mon[44017]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:04.125 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:04.125 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:04.182 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:05.184 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:05.304 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:05.528 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:05.528 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:05.566 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:05 trial183 ceph-mon[43553]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:05 trial183 ceph-mon[43553]: pgmap v71: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 72 op/s 2026-02-24T16:42:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:05 trial151 ceph-mon[39453]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:05 trial151 ceph-mon[39453]: pgmap v71: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 72 op/s 2026-02-24T16:42:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:05 trial153 ceph-mon[44017]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:05 trial153 ceph-mon[44017]: pgmap v71: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 72 op/s 2026-02-24T16:42:06.567 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:06.686 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:06.901 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:06 trial151 ceph-mon[39453]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:06.901 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:06.901 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:06.938 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:06.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:06 trial183 ceph-mon[43553]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:07.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:06 trial153 ceph-mon[44017]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:07.940 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:07.964 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:07 trial151 ceph-mon[39453]: pgmap v72: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 46 op/s 2026-02-24T16:42:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:07 trial183 ceph-mon[43553]: pgmap v72: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 46 op/s 2026-02-24T16:42:08.056 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:08.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:07 trial153 ceph-mon[44017]: pgmap v72: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 46 op/s 2026-02-24T16:42:08.272 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:08.272 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:08.309 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:08.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:08 trial183 ceph-mon[43553]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:08.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:08 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:08.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:08 trial151 ceph-mon[39453]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:08.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:08 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:09.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:08 trial153 ceph-mon[44017]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:09.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:08 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:09.310 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:09.427 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:09.646 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:09.646 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:09.683 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:09 trial183 ceph-mon[43553]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:09 trial183 ceph-mon[43553]: pgmap v73: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.2 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2026-02-24T16:42:09.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:09 trial151 ceph-mon[39453]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:09.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:09 trial151 ceph-mon[39453]: pgmap v73: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.2 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2026-02-24T16:42:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:09 trial153 ceph-mon[44017]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:09 trial153 ceph-mon[44017]: pgmap v73: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.2 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2026-02-24T16:42:10.684 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:10.800 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:10.915 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:10 trial151 ceph-mon[39453]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:10.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:10 trial183 ceph-mon[43553]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:11.016 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:11.016 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:11.053 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:11.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:10 trial153 ceph-mon[44017]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:11 trial183 ceph-mon[43553]: pgmap v74: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 15 op/s 2026-02-24T16:42:11.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:11 trial151 ceph-mon[39453]: pgmap v74: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 15 op/s 2026-02-24T16:42:12.053 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:11 trial153 ceph-mon[44017]: pgmap v74: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 15 op/s 2026-02-24T16:42:12.171 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:12.387 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:12.387 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:12.425 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:12.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:12 trial151 ceph-mon[39453]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:12.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:12 trial183 ceph-mon[43553]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:13.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:12 trial153 ceph-mon[44017]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:13.426 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:13.544 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:13.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:13 trial151 ceph-mon[39453]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:13.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:13 trial151 ceph-mon[39453]: pgmap v75: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:13.759 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:13.760 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:13.797 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:13 trial183 ceph-mon[43553]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:13 trial183 ceph-mon[43553]: pgmap v75: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:13 trial153 ceph-mon[44017]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:13 trial153 ceph-mon[44017]: pgmap v75: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:14.798 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:14.916 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:14.936 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:14 trial151 ceph-mon[39453]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:14.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:14 trial183 ceph-mon[43553]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:15.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:14 trial153 ceph-mon[44017]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:15.127 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:15.127 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:15.163 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:15 trial183 ceph-mon[43553]: pgmap v76: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:15.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:15 trial151 ceph-mon[39453]: pgmap v76: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:16.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:15 trial153 ceph-mon[44017]: pgmap v76: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:16.164 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:16.283 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:16.502 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:16.502 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:16.539 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:16 trial183 ceph-mon[43553]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:16.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:16 trial151 ceph-mon[39453]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:17.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:16 trial153 ceph-mon[44017]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:17.541 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:17.657 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:17.871 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:17.871 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:17.872 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:17 trial151 ceph-mon[39453]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:17.872 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:17 trial151 ceph-mon[39453]: pgmap v77: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:17.907 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:17.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:17 trial183 ceph-mon[43553]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:17.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:17 trial183 ceph-mon[43553]: pgmap v77: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:18.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:17 trial153 ceph-mon[44017]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:17 trial153 ceph-mon[44017]: pgmap v77: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:18.908 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:19.024 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:19.238 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:19.238 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:19.275 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:19 trial183 ceph-mon[43553]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:19 trial183 ceph-mon[43553]: pgmap v78: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:19.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:19 trial151 ceph-mon[39453]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:19.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:19 trial151 ceph-mon[39453]: pgmap v78: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:19 trial153 ceph-mon[44017]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:19 trial153 ceph-mon[44017]: pgmap v78: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:20.276 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:20.395 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:20.613 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:20.613 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:20.651 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:20.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:20 trial183 ceph-mon[43553]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:20.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:20 trial151 ceph-mon[39453]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:21.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:20 trial153 ceph-mon[44017]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:21.653 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:21.770 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:21.983 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:21.983 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:21.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:21 trial151 ceph-mon[39453]: pgmap v79: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:21.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:21 trial151 ceph-mon[39453]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:21.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:21 trial183 ceph-mon[43553]: pgmap v79: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:21.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:21 trial183 ceph-mon[43553]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:22.037 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:22.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:21 trial153 ceph-mon[44017]: pgmap v79: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:22.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:21 trial153 ceph-mon[44017]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:23.038 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:23.155 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:23.374 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:23.374 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:23.411 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:23 trial183 ceph-mon[43553]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:23 trial183 ceph-mon[43553]: pgmap v80: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:23 trial151 ceph-mon[39453]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:23.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:23 trial151 ceph-mon[39453]: pgmap v80: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:23 trial153 ceph-mon[44017]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:23 trial153 ceph-mon[44017]: pgmap v80: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:24.411 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:24.528 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:24.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:24 trial151 ceph-mon[39453]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:24.751 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:24.751 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:24.789 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:24.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:24 trial183 ceph-mon[43553]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:25.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:24 trial153 ceph-mon[44017]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:25.790 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:25.909 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:25.931 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:25 trial151 ceph-mon[39453]: pgmap v81: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:25.932 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:25 trial151 ceph-mon[39453]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:25 trial183 ceph-mon[43553]: pgmap v81: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:25 trial183 ceph-mon[43553]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:25 trial153 ceph-mon[44017]: pgmap v81: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:25 trial153 ceph-mon[44017]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:26.132 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:26.132 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:26.169 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:27.170 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:27.290 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:27.505 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:27.505 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:41:54.921547Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:27.544 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:27 trial183 ceph-mon[43553]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:27 trial183 ceph-mon[43553]: pgmap v82: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:27.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:27 trial151 ceph-mon[39453]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:27.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:27 trial151 ceph-mon[39453]: pgmap v82: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:27 trial153 ceph-mon[44017]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:27 trial153 ceph-mon[44017]: pgmap v82: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:28.477 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 bash[52565]: Copying config sha256:a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 2026-02-24T16:42:28.477 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 bash[52565]: Writing manifest to image destination 2026-02-24T16:42:28.477 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 podman[52565]: 2026-02-24 16:42:28.272244909 +0000 UTC m=+32.798557770 container create 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b, maintainer=Ceph Developers , io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.git.branch=devel, io.spdk.git.branch=undefined, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, io.spdk.build.ceph-release=20.2.0, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.buildah.version=1.29.0, url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, vendor=Red Hat, Inc., io.openshift.wants=ceph,rbd, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.ceph.maintainer=Ceph Developers , distribution-scope=public, io.ceph.nvmeof=, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.repo=https://github.com/ceph/spdk.git, com.redhat.component=ubi9-container, io.ceph.version=1.7.1, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.build.makeflags=, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, build-date=2026-02-24 13:05:47 UTC, io.spdk.build.pkgdep-args=--rbd, org.centos.version=9.0-30.el9, release=, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.version=25.09, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.component=ceph-nvmeof, architecture=x86_64, name=ceph-nvmeof, version=1.7.1) 2026-02-24T16:42:28.478 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 podman[52565]: 2026-02-24 16:42:28.306701195 +0000 UTC m=+32.833014066 container init 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b, distribution-scope=public, io.ceph.summary=Ceph NVMe over Fabrics Gateway, com.redhat.component=ubi9-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.maintainer=Ceph Developers , version=1.7.1, io.openshift.wants=ceph,rbd, io.ceph.component=ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, name=ceph-nvmeof, io.buildah.version=1.29.0, release=, vendor=Red Hat, Inc., io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.repo=https://github.com/ceph/spdk.git, maintainer=Ceph Developers , io.spdk.build.ceph-release=20.2.0, io.spdk.git.branch=undefined, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, org.centos.version=9.0-30.el9, io.spdk.build.pkgdep-args=--rbd, vcs-type=git, io.spdk.version=25.09, io.ceph.git.branch=devel, build-date=2026-02-24 13:05:47 UTC, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.version=1.7.1, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, url=https://github.com/ceph/ceph-nvmeof, architecture=x86_64, io.ceph.nvmeof=, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, io.spdk.build.makeflags=) 2026-02-24T16:42:28.478 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 podman[52565]: 2026-02-24 16:42:28.309163321 +0000 UTC m=+32.835476192 container start 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.git.branch=devel, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, release=, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.version=1.7.1, io.ceph.nvmeof=, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.makeflags=, io.spdk.build.pkgdep-args=--rbd, com.redhat.component=ubi9-container, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, maintainer=Ceph Developers , io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.ceph-release=20.2.0, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.maintainer=Ceph Developers , org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.version=25.09, build-date=2026-02-24 13:05:47 UTC, io.ceph.component=ceph-nvmeof, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, vcs-type=git, org.centos.version=9.0-30.el9, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=x86-64-v2, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, name=ceph-nvmeof, version=1.7.1, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.29.0, io.spdk.git.branch=undefined, architecture=x86_64, io.openshift.wants=ceph,rbd, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.repo=https://github.com/ceph/spdk.git) 2026-02-24T16:42:28.478 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 bash[52565]: 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf 2026-02-24T16:42:28.478 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 podman[52565]: 2026-02-24 16:42:28.265603245 +0000 UTC m=+32.791916116 image pull a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 quay.io/ceph/nvmeof:devel 2026-02-24T16:42:28.478 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 systemd[1]: Started Ceph nvmeof.nvmeof.b for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:42:28.545 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:28.660 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:28.798 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:28 trial191 systemd[1]: Starting Ceph nvmeof.nvmeof.d for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:42:28.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:28.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:28.819 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO utils.py:421 (2): Initialize gateway log level to "INFO" 2026-02-24T16:42:28.819 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO utils.py:436 (2): Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.nvmeof.b, using rotation 2026-02-24T16:42:28.819 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:86 (2): Using NVMeoF gateway version 1.7.1 2026-02-24T16:42:28.819 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:89 (2): Configured SPDK version 25.09 2026-02-24T16:42:28.819 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:92 (2): Using vstart cluster version based on 20.2.0 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:95 (2): NVMeoF gateway built on: 2026-02-24 13:05:47 UTC 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:98 (2): NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:101 (2): NVMeoF gateway Git branch: devel 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:104 (2): NVMeoF gateway Git commit: 5b3adc3ce6aa283baaa0c72d4265014805a14f03 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:110 (2): SPDK Git repository: https://github.com/ceph/spdk.git 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:113 (2): SPDK Git branch: undefined 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:116 (2): SPDK Git commit: cf5921356df981b1954fa7e2fefecb1c3242c8f9 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:65 (2): Using configuration file /src/ceph-nvmeof.conf 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:67 (2): ====================================== Configuration file content ====================================== 2026-02-24T16:42:28.820 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This file is generated by cephadm. 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [gateway] 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): name = client.nvmeof.nvmeof.b 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): group = mygroup0 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): addr = 10.20.193.153 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): port = 5500 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): enable_auth = False 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): state_update_notify = True 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): state_update_interval_sec = 5 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): break_update_interval_sec = 25 2026-02-24T16:42:28.821 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): enable_spdk_discovery_controller = False 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): encryption_key = /encryption.key 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): rebalance_period_sec = 7 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_gws_in_grp = 16 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_ns_to_change_lb_grp = 8 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): enable_prometheus_exporter = True 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_exporter_ssl = False 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_port = 10008 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_stats_interval = 10 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_frequency_slow_down_factor = 3.0 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_cycles_to_adjust_speed = 3 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_startup_delay = 240 2026-02-24T16:42:28.822 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): prometheus_connection_list_cache_expiration = 60 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): verify_nqns = True 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): verify_keys = True 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): verify_listener_ip = True 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): abort_on_update_error = True 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_ignore_unlock_errors = False 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.823 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_lock_on_read = True 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_lock_duration = 40 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_lock_retries = 30 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_lock_retry_sleep_interval = 1.0 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_update_reloads = 10 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): omap_file_update_attempts = 500 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): allowed_consecutive_spdk_ping_failures = 1 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): spdk_ping_interval_in_seconds = 2.0 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): ping_spdk_under_lock = False 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): enable_monitor_client = True 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_hosts_per_namespace = 16 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_namespaces_with_netmask = 1000 2026-02-24T16:42:28.824 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_subsystems = 128 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_hosts = 2048 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_namespaces = 4096 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_namespaces_per_subsystem = 512 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_hosts_per_subsystem = 128 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): subsystem_cache_expiration = 30 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): force_tls = False 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_message_length_in_mb = 4 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): io_stats_enabled = True 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [gateway-logs] 2026-02-24T16:42:28.825 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): log_level = INFO 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): log_files_enabled = True 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): log_files_rotation_enabled = True 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): verbose_log_messages = True 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_log_file_size_in_mb = 10 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_log_files_count = 20 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): max_log_directory_backups = 10 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): log_directory = /var/log/ceph/ 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [discovery] 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): addr = 10.20.193.153 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): port = 8009 2026-02-24T16:42:28.826 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): bind_retries_limit = 10 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): bind_sleep_interval = 0.5 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [ceph] 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): pool = mypool 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): config_file = /etc/ceph/ceph.conf 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): id = nvmeof.nvmeof.b 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [mtls] 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): server_key = /server.key 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): client_key = /client.key 2026-02-24T16:42:28.827 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): server_cert = /server.cert 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): client_cert = /client.cert 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): root_ca_cert = /root.ca.cert 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [spdk] 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): tgt_path = /usr/local/bin/nvmf_tgt 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): rpc_socket_dir = /var/tmp/ 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): rpc_socket_name = spdk.sock 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): timeout = 60.0 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): cluster_connections = 32 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): protocol_log_level = WARNING 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): conn_retries = 10 2026-02-24T16:42:28.828 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): transports = tcp 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7} 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): enable_dsa_acceleration = False 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): rbd_with_crc32c = True 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): qos_timeslice_in_usecs = 0 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): notifications_interval = 60 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): [monitor] 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:72 (2): timeout = 1.0 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO config.py:73 (2): ======================================================================================================== 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] WARNING server.py:151 (2): No valid encryption key file was set. Any attempt to encrypt or decrypt keys would fail 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:163 (2): Starting gateway client.nvmeof.nvmeof.b 2026-02-24T16:42:28.829 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:297 (2): Starting serve, monitor client version: ceph version 20.3.0-5497-gcc64fab7 (cc64fab70a7461f69d05383a379e7f58ec56db05) tentacle (dev - RelWithDebInfo) 2026-02-24T16:42:28.830 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO state.py:891 (2): First gateway: created object nvmeof.mygroup0.state 2026-02-24T16:42:28.830 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:897 (2): DSA acceleration device configuration is disabled 2026-02-24T16:42:28.830 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:424 (2): Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.b --gateway-address 10.20.193.153:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 10.20.193.153:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.b -k /etc/ceph/keyring 2026-02-24T16:42:28.830 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:430 (2): monitor client process id: 19 2026-02-24T16:42:28.830 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:28] INFO server.py:264 (2): MonitorGroup server is listening on 10.20.193.153:5499 for group id 2026-02-24T16:42:28.879 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:28.879 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:28.349544Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:28.879 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:28 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:28.916 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:28.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:28.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:28.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:28.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:28.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:28 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:28 trial191 bash[55771]: Trying to pull quay.io/ceph/nvmeof:devel... 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Getting image source signatures 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:97c2269eda53f9b2711c2f8d2a8e46a7596573b00d87ac66af98df9a4aa230df 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:92f0847c2c039b300b3cf205eb0d401fa4cd1996dcf12e21a355ef976109a3e4 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:2349b23d12c3e608eb56237429d4d7de191bafc617a6e5e4c03c1bc94077bb03 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:e25d39bb0eba21d924660760f0b2d693bab4ca0186b73367fd85c204c3dc43e4 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:b730aed13e4a7b32d77ec274848833027c1999a6a171c890216e7b49c88c19df 2026-02-24T16:42:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:b303585155d1d10cddccb02205e343a83ca161a6ff562534866a28a443e9bdc0 2026-02-24T16:42:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:29 trial191 bash[55771]: Copying blob sha256:10c94e981653bc74ae721b67f606770aae6a996ea71c75d6c6bc5c6cc0740776 2026-02-24T16:42:29.917 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:29.941 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:29 trial151 ceph-mon[39453]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:29.941 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:29 trial151 ceph-mon[39453]: gateway address: 10.20.193.191 from map_addr=None spec.addr=None host_ip='10.20.193.191' 2026-02-24T16:42:29.941 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:29 trial151 ceph-mon[39453]: discovery address: 10.20.193.191 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.191' 2026-02-24T16:42:29.941 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:29 trial151 ceph-mon[39453]: Deploying daemon nvmeof.nvmeof.d on trial191 2026-02-24T16:42:29.942 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:29 trial151 ceph-mon[39453]: pgmap v83: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:29 trial183 ceph-mon[43553]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:29 trial183 ceph-mon[43553]: gateway address: 10.20.193.191 from map_addr=None spec.addr=None host_ip='10.20.193.191' 2026-02-24T16:42:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:29 trial183 ceph-mon[43553]: discovery address: 10.20.193.191 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.191' 2026-02-24T16:42:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:29 trial183 ceph-mon[43553]: Deploying daemon nvmeof.nvmeof.d on trial191 2026-02-24T16:42:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:29 trial183 ceph-mon[43553]: pgmap v83: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:30.035 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:29 trial153 ceph-mon[44017]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:29 trial153 ceph-mon[44017]: gateway address: 10.20.193.191 from map_addr=None spec.addr=None host_ip='10.20.193.191' 2026-02-24T16:42:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:29 trial153 ceph-mon[44017]: discovery address: 10.20.193.191 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.191' 2026-02-24T16:42:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:29 trial153 ceph-mon[44017]: Deploying daemon nvmeof.nvmeof.d on trial191 2026-02-24T16:42:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:29 trial153 ceph-mon[44017]: pgmap v83: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:30.253 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:30.253 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:28.349544Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:30.289 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:30 trial183 ceph-mon[43553]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:30.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:30 trial151 ceph-mon[39453]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:31.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:30 trial153 ceph-mon[44017]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:31.290 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:31.406 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:31.623 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:31.623 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:28.349544Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:31.659 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:31.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:31 trial183 ceph-mon[43553]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:31.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:31 trial183 ceph-mon[43553]: pgmap v84: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:31.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:31 trial151 ceph-mon[39453]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:31.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:31 trial151 ceph-mon[39453]: pgmap v84: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:32.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:31 trial153 ceph-mon[44017]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:32.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:31 trial153 ceph-mon[44017]: pgmap v84: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:32.660 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:32.778 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:32.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:32 trial183 ceph-mon[43553]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:32.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:32 trial151 ceph-mon[39453]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:32.991 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:32.991 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:32.879191Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:33.006 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 bash[55771]: Copying config sha256:a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 2026-02-24T16:42:33.006 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 bash[55771]: Writing manifest to image destination 2026-02-24T16:42:33.006 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 podman[55771]: 2026-02-24 16:42:32.806207467 +0000 UTC m=+3.935311828 container create 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.maintainer=Ceph Developers , io.spdk.build.target-arch=x86-64-v2, io.ceph.version=1.7.1, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.wants=ceph,rbd, summary=Ceph NVMe over Fabrics Gateway, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, vcs-type=git, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., version=1.7.1, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, org.centos.version=9.0-30.el9, build-date=2026-02-24 13:05:47 UTC, io.spdk.build.pkgdep-args=--rbd, distribution-scope=public, name=ceph-nvmeof, io.spdk.git.branch=undefined, io.ceph.git.branch=devel, io.spdk.version=25.09, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, url=https://github.com/ceph/ceph-nvmeof, com.redhat.component=ubi9-container, release=, io.spdk.build.ceph-release=20.2.0, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, vendor=Red Hat, Inc., architecture=x86_64, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.buildah.version=1.29.0, io.spdk.build.makeflags=, io.ceph.component=ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.nvmeof=, maintainer=Ceph Developers ) 2026-02-24T16:42:33.007 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 podman[55771]: 2026-02-24 16:42:32.83874229 +0000 UTC m=+3.967846661 container init 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.git.repo=https://github.com/ceph/spdk.git, name=ceph-nvmeof, io.ceph.git.branch=devel, com.redhat.component=ubi9-container, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, architecture=x86_64, io.spdk.git.branch=undefined, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.build.ceph-release=20.2.0, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, distribution-scope=public, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.maintainer=Ceph Developers , version=1.7.1, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=x86-64-v2, vcs-type=git, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.version=25.09, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.wants=ceph,rbd, io.buildah.version=1.29.0, build-date=2026-02-24 13:05:47 UTC, vendor=Red Hat, Inc., release=, io.ceph.version=1.7.1, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, org.centos.version=9.0-30.el9, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.pkgdep-args=--rbd, io.ceph.nvmeof=, maintainer=Ceph Developers , io.spdk.build.makeflags=) 2026-02-24T16:42:33.007 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 podman[55771]: 2026-02-24 16:42:32.840921526 +0000 UTC m=+3.970025887 container start 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d, version=1.7.1, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, name=ceph-nvmeof, io.spdk.git.branch=undefined, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.openshift.wants=ceph,rbd, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Ceph NVMe over Fabrics Gateway, io.ceph.nvmeof=, maintainer=Ceph Developers , io.spdk.build.ceph-release=20.2.0, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, url=https://github.com/ceph/ceph-nvmeof, com.redhat.component=ubi9-container, io.spdk.build.pkgdep-args=--rbd, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.component=ceph-nvmeof, architecture=x86_64, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.version=1.7.1, vcs-type=git, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2026-02-24 13:05:47 UTC, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, vendor=Red Hat, Inc., distribution-scope=public, org.centos.version=9.0-30.el9, release=, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.maintainer=Ceph Developers , io.ceph.url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.makeflags=, io.spdk.version=25.09, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.git.branch=devel, io.buildah.version=1.29.0, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2026-02-24T16:42:33.007 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 bash[55771]: 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 2026-02-24T16:42:33.007 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 podman[55771]: 2026-02-24 16:42:32.800130621 +0000 UTC m=+3.929234982 image pull a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 quay.io/ceph/nvmeof:devel 2026-02-24T16:42:33.007 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:32 trial191 systemd[1]: Started Ceph nvmeof.nvmeof.d for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:42:33.030 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:33.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:32 trial153 ceph-mon[44017]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO utils.py:421 (2): Initialize gateway log level to "INFO" 2026-02-24T16:42:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO utils.py:436 (2): Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.nvmeof.d, using rotation 2026-02-24T16:42:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:86 (2): Using NVMeoF gateway version 1.7.1 2026-02-24T16:42:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:89 (2): Configured SPDK version 25.09 2026-02-24T16:42:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:92 (2): Using vstart cluster version based on 20.2.0 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:95 (2): NVMeoF gateway built on: 2026-02-24 13:05:47 UTC 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:98 (2): NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:101 (2): NVMeoF gateway Git branch: devel 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:104 (2): NVMeoF gateway Git commit: 5b3adc3ce6aa283baaa0c72d4265014805a14f03 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:110 (2): SPDK Git repository: https://github.com/ceph/spdk.git 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:113 (2): SPDK Git branch: undefined 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:116 (2): SPDK Git commit: cf5921356df981b1954fa7e2fefecb1c3242c8f9 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:65 (2): Using configuration file /src/ceph-nvmeof.conf 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:67 (2): ====================================== Configuration file content ====================================== 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This file is generated by cephadm. 2026-02-24T16:42:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [gateway] 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): name = client.nvmeof.nvmeof.d 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): group = mygroup0 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): addr = 10.20.193.191 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): port = 5500 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): enable_auth = False 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): state_update_notify = True 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): state_update_interval_sec = 5 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): break_update_interval_sec = 25 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): enable_spdk_discovery_controller = False 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): encryption_key = /encryption.key 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): rebalance_period_sec = 7 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_gws_in_grp = 16 2026-02-24T16:42:33.343 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_ns_to_change_lb_grp = 8 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): enable_prometheus_exporter = True 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_exporter_ssl = False 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_port = 10008 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_stats_interval = 10 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_frequency_slow_down_factor = 3.0 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_cycles_to_adjust_speed = 3 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_startup_delay = 240 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): prometheus_connection_list_cache_expiration = 60 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): verify_nqns = True 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): verify_keys = True 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): verify_listener_ip = True 2026-02-24T16:42:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.345 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:33.345 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.345 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): abort_on_update_error = True 2026-02-24T16:42:33.345 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.345 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_ignore_unlock_errors = False 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_lock_on_read = True 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_lock_duration = 40 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_lock_retries = 30 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_lock_retry_sleep_interval = 1.0 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_update_reloads = 10 2026-02-24T16:42:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): omap_file_update_attempts = 500 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): allowed_consecutive_spdk_ping_failures = 1 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): spdk_ping_interval_in_seconds = 2.0 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): ping_spdk_under_lock = False 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): enable_monitor_client = True 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_hosts_per_namespace = 16 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_namespaces_with_netmask = 1000 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_subsystems = 128 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_hosts = 2048 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_namespaces = 4096 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_namespaces_per_subsystem = 512 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_hosts_per_subsystem = 128 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): subsystem_cache_expiration = 30 2026-02-24T16:42:33.347 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): force_tls = False 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_message_length_in_mb = 4 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): io_stats_enabled = True 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [gateway-logs] 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): log_level = INFO 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): log_files_enabled = True 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): log_files_rotation_enabled = True 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): verbose_log_messages = True 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_log_file_size_in_mb = 10 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_log_files_count = 20 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): max_log_directory_backups = 10 2026-02-24T16:42:33.348 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): log_directory = /var/log/ceph/ 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [discovery] 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): addr = 10.20.193.191 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): port = 8009 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): bind_retries_limit = 10 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): bind_sleep_interval = 0.5 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [ceph] 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): pool = mypool 2026-02-24T16:42:33.349 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): config_file = /etc/ceph/ceph.conf 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): id = nvmeof.nvmeof.d 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [mtls] 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): server_key = /server.key 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): client_key = /client.key 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): server_cert = /server.cert 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): client_cert = /client.cert 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): root_ca_cert = /root.ca.cert 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [spdk] 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): tgt_path = /usr/local/bin/nvmf_tgt 2026-02-24T16:42:33.350 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): rpc_socket_dir = /var/tmp/ 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): rpc_socket_name = spdk.sock 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): timeout = 60.0 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): cluster_connections = 32 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): protocol_log_level = WARNING 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): conn_retries = 10 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): transports = tcp 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7} 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): enable_dsa_acceleration = False 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): rbd_with_crc32c = True 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): qos_timeslice_in_usecs = 0 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): notifications_interval = 60 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): 2026-02-24T16:42:33.351 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): [monitor] 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:72 (2): timeout = 1.0 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO config.py:73 (2): ======================================================================================================== 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] WARNING server.py:151 (2): No valid encryption key file was set. Any attempt to encrypt or decrypt keys would fail 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:163 (2): Starting gateway client.nvmeof.nvmeof.d 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:297 (2): Starting serve, monitor client version: ceph version 20.3.0-5497-gcc64fab7 (cc64fab70a7461f69d05383a379e7f58ec56db05) tentacle (dev - RelWithDebInfo) 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO state.py:894 (2): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:897 (2): DSA acceleration device configuration is disabled 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:424 (2): Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.d --gateway-address 10.20.193.191:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 10.20.193.191:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.d -k /etc/ceph/keyring 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:430 (2): monitor client process id: 19 2026-02-24T16:42:33.352 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:33] INFO server.py:264 (2): MonitorGroup server is listening on 10.20.193.191:5499 for group id 2026-02-24T16:42:33.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:33 trial183 bash[56386]: Trying to pull quay.io/ceph/nvmeof:devel... 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: pgmap v85: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:33 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: pgmap v85: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:33 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:34.031 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: pgmap v85: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:33 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:34.145 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:34.365 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:34.365 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:32.879191Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:34.401 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:34 trial183 ceph-mon[43553]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:34 trial183 ceph-mon[43553]: gateway address: 10.20.193.183 from map_addr=None spec.addr=None host_ip='10.20.193.183' 2026-02-24T16:42:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:34 trial183 ceph-mon[43553]: discovery address: 10.20.193.183 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.183' 2026-02-24T16:42:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:34 trial183 ceph-mon[43553]: Deploying daemon nvmeof.nvmeof.c on trial183 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:34 trial183 ceph-mon[43553]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Getting image source signatures 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:92f0847c2c039b300b3cf205eb0d401fa4cd1996dcf12e21a355ef976109a3e4 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:2349b23d12c3e608eb56237429d4d7de191bafc617a6e5e4c03c1bc94077bb03 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:e25d39bb0eba21d924660760f0b2d693bab4ca0186b73367fd85c204c3dc43e4 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:97c2269eda53f9b2711c2f8d2a8e46a7596573b00d87ac66af98df9a4aa230df 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:b730aed13e4a7b32d77ec274848833027c1999a6a171c890216e7b49c88c19df 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:b303585155d1d10cddccb02205e343a83ca161a6ff562534866a28a443e9bdc0 2026-02-24T16:42:34.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:34 trial183 bash[56386]: Copying blob sha256:10c94e981653bc74ae721b67f606770aae6a996ea71c75d6c6bc5c6cc0740776 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:34 trial151 ceph-mon[39453]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:34 trial151 ceph-mon[39453]: gateway address: 10.20.193.183 from map_addr=None spec.addr=None host_ip='10.20.193.183' 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:34 trial151 ceph-mon[39453]: discovery address: 10.20.193.183 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.183' 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:34 trial151 ceph-mon[39453]: Deploying daemon nvmeof.nvmeof.c on trial183 2026-02-24T16:42:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:34 trial151 ceph-mon[39453]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:35.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:34 trial153 ceph-mon[44017]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:35.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:34 trial153 ceph-mon[44017]: gateway address: 10.20.193.183 from map_addr=None spec.addr=None host_ip='10.20.193.183' 2026-02-24T16:42:35.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:34 trial153 ceph-mon[44017]: discovery address: 10.20.193.183 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.183' 2026-02-24T16:42:35.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:34 trial153 ceph-mon[44017]: Deploying daemon nvmeof.nvmeof.c on trial183 2026-02-24T16:42:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:34 trial153 ceph-mon[44017]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:35.402 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:35.519 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:35.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:35 trial151 ceph-mon[39453]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:35.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:35 trial151 ceph-mon[39453]: pgmap v86: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:35.736 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:35.736 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:32.879191Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:35.773 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:35.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:35 trial183 ceph-mon[43553]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:35.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:35 trial183 ceph-mon[43553]: pgmap v86: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:36.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:35 trial153 ceph-mon[44017]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:36.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:35 trial153 ceph-mon[44017]: pgmap v86: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:36.774 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:36.892 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:36.933 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:36 trial151 ceph-mon[39453]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:36.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:36 trial183 ceph-mon[43553]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:37.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:36 trial153 ceph-mon[44017]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:37.107 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:37.107 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:32.879191Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:37.144 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:37 trial183 ceph-mon[43553]: pgmap v87: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:37.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:37 trial151 ceph-mon[39453]: pgmap v87: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:38.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:37 trial153 ceph-mon[44017]: pgmap v87: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:38.145 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:38.264 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:38.480 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:38.481 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:32.879191Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:38.519 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:38.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:38 trial151 ceph-mon[39453]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:38.806 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-mon[43553]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:38.806 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 bash[56386]: Copying config sha256:a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 bash[56386]: Writing manifest to image destination 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 podman[56386]: 2026-02-24 16:42:38.60424015 +0000 UTC m=+5.186944096 container create d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c, summary=Ceph NVMe over Fabrics Gateway, name=ceph-nvmeof, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.nvmeof=, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.makeflags=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.git.branch=devel, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., release=, io.buildah.version=1.29.0, io.spdk.build.target-arch=x86-64-v2, build-date=2026-02-24 13:05:47 UTC, url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, vendor=Red Hat, Inc., io.spdk.git.branch=undefined, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.url=https://github.com/ceph/ceph-nvmeof, distribution-scope=public, io.spdk.version=25.09, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, vcs-type=git, io.ceph.summary=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , architecture=x86_64, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.repo=https://github.com/ceph/spdk.git, com.redhat.component=ubi9-container, version=1.7.1, org.centos.version=9.0-30.el9, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.component=ceph-nvmeof, io.spdk.build.ceph-release=20.2.0, io.spdk.build.pkgdep-args=--rbd, io.ceph.version=1.7.1, io.ceph.maintainer=Ceph Developers , io.openshift.wants=ceph,rbd) 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 podman[56386]: 2026-02-24 16:42:38.598328389 +0000 UTC m=+5.181032335 image pull a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 quay.io/ceph/nvmeof:devel 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 podman[56386]: 2026-02-24 16:42:38.638563356 +0000 UTC m=+5.221267292 container init d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.ceph.url=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.ceph-release=20.2.0, io.spdk.build.target-arch=x86-64-v2, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=1.7.1, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, name=ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.git.branch=undefined, release=, io.ceph.git.branch=devel, com.redhat.component=ubi9-container, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.openshift.wants=ceph,rbd, io.spdk.build.makeflags=, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.version=1.7.1, io.ceph.nvmeof=, maintainer=Ceph Developers , io.ceph.component=ceph-nvmeof, vendor=Red Hat, Inc., io.spdk.version=25.09, org.centos.version=9.0-30.el9, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, build-date=2026-02-24 13:05:47 UTC, architecture=x86_64, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.build.pkgdep-args=--rbd, io.buildah.version=1.29.0, io.ceph.summary=Ceph NVMe over Fabrics Gateway, url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, distribution-scope=public, io.ceph.maintainer=Ceph Developers ) 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 podman[56386]: 2026-02-24 16:42:38.640941287 +0000 UTC m=+5.223645233 container start d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c, io.ceph.component=ceph-nvmeof, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.pkgdep-args=--rbd, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, org.centos.version=9.0-30.el9, io.spdk.build.ceph-release=20.2.0, version=1.7.1, io.spdk.build.makeflags=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, release=, io.buildah.version=1.29.0, io.spdk.git.repo=https://github.com/ceph/spdk.git, com.redhat.component=ubi9-container, io.spdk.git.branch=undefined, io.ceph.maintainer=Ceph Developers , build-date=2026-02-24 13:05:47 UTC, url=https://github.com/ceph/ceph-nvmeof, io.openshift.wants=ceph,rbd, vcs-type=git, io.ceph.git.branch=devel, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, architecture=x86_64, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, name=ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., summary=Ceph NVMe over Fabrics Gateway, io.ceph.version=1.7.1, io.ceph.summary=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, vendor=Red Hat, Inc., io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.nvmeof=, io.spdk.version=25.09, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 bash[56386]: d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 2026-02-24T16:42:38.807 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 systemd[1]: Started Ceph nvmeof.nvmeof.c for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:42:39.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:38 trial153 ceph-mon[44017]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO utils.py:421 (2): Initialize gateway log level to "INFO" 2026-02-24T16:42:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO utils.py:436 (2): Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c, using rotation 2026-02-24T16:42:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:86 (2): Using NVMeoF gateway version 1.7.1 2026-02-24T16:42:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:89 (2): Configured SPDK version 25.09 2026-02-24T16:42:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:92 (2): Using vstart cluster version based on 20.2.0 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:95 (2): NVMeoF gateway built on: 2026-02-24 13:05:47 UTC 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:98 (2): NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:101 (2): NVMeoF gateway Git branch: devel 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:104 (2): NVMeoF gateway Git commit: 5b3adc3ce6aa283baaa0c72d4265014805a14f03 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:110 (2): SPDK Git repository: https://github.com/ceph/spdk.git 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:113 (2): SPDK Git branch: undefined 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:116 (2): SPDK Git commit: cf5921356df981b1954fa7e2fefecb1c3242c8f9 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:65 (2): Using configuration file /src/ceph-nvmeof.conf 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:67 (2): ====================================== Configuration file content ====================================== 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This file is generated by cephadm. 2026-02-24T16:42:39.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [gateway] 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): name = client.nvmeof.nvmeof.c 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): group = mygroup0 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): addr = 10.20.193.183 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): port = 5500 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): enable_auth = False 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): state_update_notify = True 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): state_update_interval_sec = 5 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): break_update_interval_sec = 25 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): enable_spdk_discovery_controller = False 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): encryption_key = /encryption.key 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): rebalance_period_sec = 7 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_gws_in_grp = 16 2026-02-24T16:42:39.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_ns_to_change_lb_grp = 8 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): enable_prometheus_exporter = True 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_exporter_ssl = False 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_port = 10008 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_stats_interval = 10 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_frequency_slow_down_factor = 3.0 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_cycles_to_adjust_speed = 3 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_startup_delay = 240 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): prometheus_connection_list_cache_expiration = 60 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): verify_nqns = True 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): verify_keys = True 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): verify_listener_ip = True 2026-02-24T16:42:39.237 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.238 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:39.238 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.238 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): abort_on_update_error = True 2026-02-24T16:42:39.238 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.238 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_ignore_unlock_errors = False 2026-02-24T16:42:39.239 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.239 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_lock_on_read = True 2026-02-24T16:42:39.239 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_lock_duration = 40 2026-02-24T16:42:39.239 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_lock_retries = 30 2026-02-24T16:42:39.239 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_lock_retry_sleep_interval = 1.0 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_update_reloads = 10 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): omap_file_update_attempts = 500 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): allowed_consecutive_spdk_ping_failures = 1 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): spdk_ping_interval_in_seconds = 2.0 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): ping_spdk_under_lock = False 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): enable_monitor_client = True 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_hosts_per_namespace = 16 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_namespaces_with_netmask = 1000 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_subsystems = 128 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_hosts = 2048 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_namespaces = 4096 2026-02-24T16:42:39.240 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_namespaces_per_subsystem = 512 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_hosts_per_subsystem = 128 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): subsystem_cache_expiration = 30 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): force_tls = False 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_message_length_in_mb = 4 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): io_stats_enabled = True 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [gateway-logs] 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): log_level = INFO 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): log_files_enabled = True 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): log_files_rotation_enabled = True 2026-02-24T16:42:39.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): verbose_log_messages = True 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_log_file_size_in_mb = 10 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_log_files_count = 20 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): max_log_directory_backups = 10 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): log_directory = /var/log/ceph/ 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [discovery] 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): addr = 10.20.193.183 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): port = 8009 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): bind_retries_limit = 10 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): bind_sleep_interval = 0.5 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.242 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [ceph] 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): pool = mypool 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): config_file = /etc/ceph/ceph.conf 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): id = nvmeof.nvmeof.c 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [mtls] 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): server_key = /server.key 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): client_key = /client.key 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): server_cert = /server.cert 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): client_cert = /client.cert 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): root_ca_cert = /root.ca.cert 2026-02-24T16:42:39.243 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [spdk] 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): tgt_path = /usr/local/bin/nvmf_tgt 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): rpc_socket_dir = /var/tmp/ 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): rpc_socket_name = spdk.sock 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): timeout = 60.0 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): cluster_connections = 32 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): protocol_log_level = WARNING 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): conn_retries = 10 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): transports = tcp 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7} 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): enable_dsa_acceleration = False 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): rbd_with_crc32c = True 2026-02-24T16:42:39.244 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): qos_timeslice_in_usecs = 0 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): notifications_interval = 60 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): [monitor] 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:72 (2): timeout = 1.0 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO config.py:73 (2): ======================================================================================================== 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] WARNING server.py:151 (2): No valid encryption key file was set. Any attempt to encrypt or decrypt keys would fail 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:163 (2): Starting gateway client.nvmeof.nvmeof.c 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:297 (2): Starting serve, monitor client version: ceph version 20.3.0-5497-gcc64fab7 (cc64fab70a7461f69d05383a379e7f58ec56db05) tentacle (dev - RelWithDebInfo) 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO state.py:894 (2): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:897 (2): DSA acceleration device configuration is disabled 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:424 (2): Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.c --gateway-address 10.20.193.183:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 10.20.193.183:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.c -k /etc/ceph/keyring 2026-02-24T16:42:39.245 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:430 (2): monitor client process id: 19 2026-02-24T16:42:39.246 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:38] INFO server.py:264 (2): MonitorGroup server is listening on 10.20.193.183:5499 for group id 2026-02-24T16:42:39.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Trying to pull quay.io/ceph/nvmeof:devel... 2026-02-24T16:42:39.519 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:39.642 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:39.884 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:39.884 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:38.681148Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: pgmap v88: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:39.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: gateway address: 10.20.193.151 from map_addr=None spec.addr=None host_ip='10.20.193.151' 2026-02-24T16:42:39.886 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: discovery address: 10.20.193.151 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.151' 2026-02-24T16:42:39.886 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:39.886 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:39 trial151 ceph-mon[39453]: Deploying daemon nvmeof.nvmeof.a on trial151 2026-02-24T16:42:39.886 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Getting image source signatures 2026-02-24T16:42:39.886 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:92f0847c2c039b300b3cf205eb0d401fa4cd1996dcf12e21a355ef976109a3e4 2026-02-24T16:42:39.886 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:e25d39bb0eba21d924660760f0b2d693bab4ca0186b73367fd85c204c3dc43e4 2026-02-24T16:42:39.886 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:2349b23d12c3e608eb56237429d4d7de191bafc617a6e5e4c03c1bc94077bb03 2026-02-24T16:42:39.886 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:b730aed13e4a7b32d77ec274848833027c1999a6a171c890216e7b49c88c19df 2026-02-24T16:42:39.887 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:97c2269eda53f9b2711c2f8d2a8e46a7596573b00d87ac66af98df9a4aa230df 2026-02-24T16:42:39.887 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2026-02-24T16:42:39.887 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:b303585155d1d10cddccb02205e343a83ca161a6ff562534866a28a443e9bdc0 2026-02-24T16:42:39.887 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:39 trial151 bash[74357]: Copying blob sha256:10c94e981653bc74ae721b67f606770aae6a996ea71c75d6c6bc5c6cc0740776 2026-02-24T16:42:39.925 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: pgmap v88: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: gateway address: 10.20.193.151 from map_addr=None spec.addr=None host_ip='10.20.193.151' 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: discovery address: 10.20.193.151 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.151' 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:39 trial183 ceph-mon[43553]: Deploying daemon nvmeof.nvmeof.a on trial151 2026-02-24T16:42:40.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: pgmap v88: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: TLS for nvmeof service nvmeof.mypool.mygroup0 is disabled. 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: gateway address: 10.20.193.151 from map_addr=None spec.addr=None host_ip='10.20.193.151' 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: discovery address: 10.20.193.151 from map_discovery_addr=None spec.discovery_addr=None host_ip='10.20.193.151' 2026-02-24T16:42:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:40.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:39 trial153 ceph-mon[44017]: Deploying daemon nvmeof.nvmeof.a on trial151 2026-02-24T16:42:40.926 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:41.055 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:41.290 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:41.290 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:38.681148Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:41.330 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:41 trial183 ceph-mon[43553]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:41 trial183 ceph-mon[43553]: pgmap v89: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:41 trial151 ceph-mon[39453]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:41 trial151 ceph-mon[39453]: pgmap v89: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:42.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:41 trial153 ceph-mon[44017]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:42.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:41 trial153 ceph-mon[44017]: pgmap v89: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:42.330 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:42.450 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:42.686 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:42.687 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:38.681148Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:42.728 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:42.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:42 trial183 ceph-mon[43553]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:42.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:42 trial151 ceph-mon[39453]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:43.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:42 trial153 ceph-mon[44017]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:43.730 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:43.848 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:43.962 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:43 trial151 ceph-mon[39453]: pgmap v90: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:43.962 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:43 trial151 ceph-mon[39453]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:43.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:43 trial183 ceph-mon[43553]: pgmap v90: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:43.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:43 trial183 ceph-mon[43553]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:44.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:43 trial153 ceph-mon[44017]: pgmap v90: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 B/s wr, 0 op/s 2026-02-24T16:42:44.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:43 trial153 ceph-mon[44017]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:44.086 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:44.086 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:41:17.099554Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:41:17.099494Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:38.681148Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "ports": [5500, 4420, 8009, 10008], "running": 0, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:41:17.099602Z", "running": 4, "size": 4}}] 2026-02-24T16:42:44.127 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 0/4 2026-02-24T16:42:44.598 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 bash[74357]: Copying config sha256:a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 2026-02-24T16:42:44.598 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 bash[74357]: Writing manifest to image destination 2026-02-24T16:42:44.598 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 podman[74357]: 2026-02-24 16:42:44.388872487 +0000 UTC m=+5.186713332 container create ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a, release=, summary=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.buildah.version=1.29.0, architecture=x86_64, io.spdk.build.ceph-release=20.2.0, io.spdk.build.pkgdep-args=--rbd, build-date=2026-02-24 13:05:47 UTC, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., maintainer=Ceph Developers , io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.ceph.maintainer=Ceph Developers , io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.git.branch=undefined, io.openshift.wants=ceph,rbd, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, version=1.7.1, vcs-type=git, io.ceph.git.branch=devel, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.nvmeof=, io.spdk.build.makeflags=, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, distribution-scope=public, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.component=ubi9-container, io.ceph.version=1.7.1, url=https://github.com/ceph/ceph-nvmeof, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.build.target-arch=x86-64-v2, io.ceph.summary=Ceph NVMe over Fabrics Gateway, vendor=Red Hat, Inc., name=ceph-nvmeof, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.component=ceph-nvmeof, org.centos.version=9.0-30.el9, io.spdk.version=25.09) 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 podman[74357]: 2026-02-24 16:42:44.382835554 +0000 UTC m=+5.180676419 image pull a25af19136b9f2b8c372501b936fc9d2fac577330e6832d68a08b4f53d6c1f64 quay.io/ceph/nvmeof:devel 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 podman[74357]: 2026-02-24 16:42:44.42372435 +0000 UTC m=+5.221565235 container init ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=x86-64-v2, com.redhat.component=ubi9-container, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, architecture=x86_64, io.spdk.build.ceph-release=20.2.0, io.spdk.build.makeflags=, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.git.branch=undefined, maintainer=Ceph Developers , vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.build.pkgdep-args=--rbd, summary=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.nvmeof=, io.buildah.version=1.29.0, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=25.09, build-date=2026-02-24 13:05:47 UTC, io.ceph.summary=Ceph NVMe over Fabrics Gateway, version=1.7.1, io.ceph.git.branch=devel, io.openshift.wants=ceph,rbd, release=, io.ceph.version=1.7.1, io.ceph.component=ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, org.centos.version=9.0-30.el9, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vendor=Red Hat, Inc., io.ceph.maintainer=Ceph Developers , distribution-scope=public, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., name=ceph-nvmeof) 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 podman[74357]: 2026-02-24 16:42:44.426415301 +0000 UTC m=+5.224256156 container start ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., maintainer=Ceph Developers , summary=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.git.branch=devel, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.target-arch=x86-64-v2, vendor=Red Hat, Inc., io.ceph.version=1.7.1, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.maintainer=Ceph Developers , io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, architecture=x86_64, io.spdk.build.makeflags=, io.spdk.build.pkgdep-args=--rbd, org.centos.version=9.0-30.el9, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.repo=https://github.com/ceph/spdk.git, distribution-scope=public, io.spdk.git.branch=undefined, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.buildah.version=1.29.0, io.ceph.component=ceph-nvmeof, io.spdk.build.ceph-release=20.2.0, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.version=25.09, io.openshift.wants=ceph,rbd, build-date=2026-02-24 13:05:47 UTC, release=, io.ceph.nvmeof=, vcs-type=git, name=ceph-nvmeof, url=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.component=ubi9-container, version=1.7.1, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 bash[74357]: ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 systemd[1]: Started Ceph nvmeof.nvmeof.a for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:42:44.599 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO utils.py:421 (2): Initialize gateway log level to "INFO" 2026-02-24T16:42:44.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO utils.py:436 (2): Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.nvmeof.a, using rotation 2026-02-24T16:42:44.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:86 (2): Using NVMeoF gateway version 1.7.1 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:89 (2): Configured SPDK version 25.09 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:92 (2): Using vstart cluster version based on 20.2.0 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:95 (2): NVMeoF gateway built on: 2026-02-24 13:05:47 UTC 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:98 (2): NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:101 (2): NVMeoF gateway Git branch: devel 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:104 (2): NVMeoF gateway Git commit: 5b3adc3ce6aa283baaa0c72d4265014805a14f03 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:110 (2): SPDK Git repository: https://github.com/ceph/spdk.git 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:113 (2): SPDK Git branch: undefined 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:116 (2): SPDK Git commit: cf5921356df981b1954fa7e2fefecb1c3242c8f9 2026-02-24T16:42:44.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:65 (2): Using configuration file /src/ceph-nvmeof.conf 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:67 (2): ====================================== Configuration file content ====================================== 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This file is generated by cephadm. 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [gateway] 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): name = client.nvmeof.nvmeof.a 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): group = mygroup0 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): addr = 10.20.193.151 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): port = 5500 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): enable_auth = False 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): state_update_notify = True 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): state_update_interval_sec = 5 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): break_update_interval_sec = 25 2026-02-24T16:42:44.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): enable_spdk_discovery_controller = False 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): encryption_key = /encryption.key 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): rebalance_period_sec = 7 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_gws_in_grp = 16 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_ns_to_change_lb_grp = 8 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): enable_prometheus_exporter = True 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_exporter_ssl = False 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_port = 10008 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_stats_interval = 10 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_frequency_slow_down_factor = 3.0 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_cycles_to_adjust_speed = 3 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_startup_delay = 240 2026-02-24T16:42:44.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): prometheus_connection_list_cache_expiration = 60 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): verify_nqns = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): verify_keys = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): verify_listener_ip = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): abort_on_update_error = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_ignore_unlock_errors = False 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_lock_on_read = True 2026-02-24T16:42:44.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_lock_duration = 40 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_lock_retries = 30 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_lock_retry_sleep_interval = 1.0 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_update_reloads = 10 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): omap_file_update_attempts = 500 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): allowed_consecutive_spdk_ping_failures = 1 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): spdk_ping_interval_in_seconds = 2.0 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): ping_spdk_under_lock = False 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): enable_monitor_client = True 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_hosts_per_namespace = 16 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_namespaces_with_netmask = 1000 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_subsystems = 128 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_hosts = 2048 2026-02-24T16:42:44.990 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_namespaces = 4096 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_namespaces_per_subsystem = 512 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_hosts_per_subsystem = 128 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): subsystem_cache_expiration = 30 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): force_tls = False 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_message_length_in_mb = 4 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): io_stats_enabled = True 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [gateway-logs] 2026-02-24T16:42:44.991 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): log_level = INFO 2026-02-24T16:42:44.992 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): log_files_enabled = True 2026-02-24T16:42:44.992 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): log_files_rotation_enabled = True 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): verbose_log_messages = True 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_log_file_size_in_mb = 10 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_log_files_count = 20 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): max_log_directory_backups = 10 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): log_directory = /var/log/ceph/ 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [discovery] 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): addr = 10.20.193.151 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): port = 8009 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): # This is a development flag, do not change it 2026-02-24T16:42:44.993 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): abort_on_errors = True 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): bind_retries_limit = 10 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): bind_sleep_interval = 0.5 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [ceph] 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): pool = mypool 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): config_file = /etc/ceph/ceph.conf 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): id = nvmeof.nvmeof.a 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [mtls] 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): server_key = /server.key 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): client_key = /client.key 2026-02-24T16:42:44.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): server_cert = /server.cert 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): client_cert = /client.cert 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): root_ca_cert = /root.ca.cert 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [spdk] 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): tgt_path = /usr/local/bin/nvmf_tgt 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): rpc_socket_dir = /var/tmp/ 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): rpc_socket_name = spdk.sock 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): timeout = 60.0 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): cluster_connections = 32 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): protocol_log_level = WARNING 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): conn_retries = 10 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): transports = tcp 2026-02-24T16:42:44.995 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7} 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): enable_dsa_acceleration = False 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): rbd_with_crc32c = True 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): qos_timeslice_in_usecs = 0 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): notifications_interval = 60 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): [monitor] 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:72 (2): timeout = 1.0 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO config.py:73 (2): ======================================================================================================== 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] WARNING server.py:151 (2): No valid encryption key file was set. Any attempt to encrypt or decrypt keys would fail 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:163 (2): Starting gateway client.nvmeof.nvmeof.a 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:297 (2): Starting serve, monitor client version: ceph version 20.3.0-5497-gcc64fab7 (cc64fab70a7461f69d05383a379e7f58ec56db05) tentacle (dev - RelWithDebInfo) 2026-02-24T16:42:44.996 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO state.py:894 (2): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:44.997 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:897 (2): DSA acceleration device configuration is disabled 2026-02-24T16:42:44.997 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:424 (2): Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.a --gateway-address 10.20.193.151:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 10.20.193.151:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.a -k /etc/ceph/keyring 2026-02-24T16:42:44.997 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:430 (2): monitor client process id: 19 2026-02-24T16:42:44.997 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:44] INFO server.py:264 (2): MonitorGroup server is listening on 10.20.193.151:5499 for group id 2026-02-24T16:42:45.127 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch ls -f json --refresh 2026-02-24T16:42:45.245 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:42:45.460 INFO:teuthology.orchestra.run.trial151.stdout: 2026-02-24T16:42:45.460 INFO:teuthology.orchestra.run.trial151.stdout:[{"events": ["2026-02-24T16:41:00.571830Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["trial151=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2026-02-24T16:41:00.565930Z", "last_refresh": "2026-02-24T16:42:45.013781Z", "running": 1, "size": 1}}, {"events": ["2026-02-24T16:40:52.975430Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["trial151:10.20.193.151=a", "trial153:10.20.193.153=b", "trial183:10.20.193.183=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2026-02-24T16:40:44.836019Z", "last_refresh": "2026-02-24T16:42:44.986950Z", "running": 3, "size": 3}}, {"events": ["2026-02-24T16:42:44.474519Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["trial151=nvmeof.a", "trial153=nvmeof.b", "trial183=nvmeof.c", "trial191=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"abort_discovery_on_errors": true, "abort_on_errors": true, "abort_on_update_error": true, "allowed_consecutive_spdk_ping_failures": 1, "break_update_interval_sec": 25, "cluster_connections": 32, "conn_retries": 10, "discovery_bind_retries_limit": 10, "discovery_bind_sleep_interval": 0.5, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "io_stats_enabled": true, "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_gws_in_grp": 16, "max_hosts": 2048, "max_hosts_per_namespace": 16, "max_hosts_per_subsystem": 128, "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "max_message_length_in_mb": 4, "max_namespaces": 4096, "max_namespaces_per_subsystem": 512, "max_namespaces_with_netmask": 1000, "max_ns_to_change_lb_grp": 8, "max_subsystems": 128, "monitor_timeout": 1.0, "notifications_interval": 60, "omap_file_lock_duration": 40, "omap_file_lock_on_read": true, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_attempts": 500, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "prometheus_connection_list_cache_expiration": 60, "prometheus_cycles_to_adjust_speed": 3, "prometheus_frequency_slow_down_factor": 3.0, "prometheus_port": 10008, "prometheus_startup_delay": 240, "prometheus_stats_interval": 10, "rbd_with_crc32c": true, "rebalance_period_sec": 7, "rpc_socket_dir": "/var/tmp/", "rpc_socket_name": "spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "subsystem_cache_expiration": 30, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_keys": true, "verify_listener_ip": true, "verify_nqns": true}, "status": {"created": "2026-02-24T16:41:54.919142Z", "last_refresh": "2026-02-24T16:42:44.961289Z", "ports": [5500, 4420, 8009, 10008], "running": 4, "size": 4}}, {"events": ["2026-02-24T16:41:29.193299Z service:osd.default [INFO] \"service was created\""], "placement": {"host_pattern": "trial191"}, "service_id": "default", "service_name": "osd.default", "service_type": "osd", "spec": {"data_devices": {"paths": ["vg_nvme/lv_4"]}, "filter_logic": "AND", "objectstore": "bluestore", "osd_type": "classic", "termination_grace_period_seconds": 30}, "status": {"created": "2026-02-24T16:41:29.192108Z", "last_refresh": "2026-02-24T16:42:44.961219Z", "running": 4, "size": 4}}] 2026-02-24T16:42:45.506 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 4/4 2026-02-24T16:42:45.506 INFO:teuthology.run_tasks:Running task workunit... 2026-02-24T16:42:45.532 INFO:tasks.workunit:Pulling workunits from ref 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:42:45.533 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-24T16:42:45.533 DEBUG:teuthology.orchestra.run.trial051:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:42:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:42:45.551 INFO:teuthology.orchestra.run.trial051.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-24T16:42:45.551 DEBUG:teuthology.orchestra.run.trial051:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:42:45.608 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-24T16:42:45.608 DEBUG:teuthology.orchestra.run.trial051:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-24T16:42:45.663 INFO:tasks.workunit:timeout=3h 2026-02-24T16:42:45.663 INFO:tasks.workunit:cleanup=True 2026-02-24T16:42:45.663 DEBUG:teuthology.orchestra.run.trial051:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:42:45.727 INFO:tasks.workunit.client.0.trial051.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:42:45.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: pgmap v91: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: Health check failed: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway. (NVMEOF_SINGLE_GATEWAY) 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:45.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:45 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:45.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:253 (2): Gateway client.nvmeof.nvmeof.a group id=0 2026-02-24T16:42:45.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:268 (2): Stopping the MonitorGroup server... 2026-02-24T16:42:45.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:271 (2): The MonitorGroup gRPC server has stopped... 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:576 (2): SPDK Target Path: /usr/local/bin/nvmf_tgt 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:593 (2): SPDK Socket: /var/tmp/spdk.sock 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:612 (2): SPDK will use huge pages, probing... 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:1022 (2): Requested huge pages count is 2048 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:1042 (2): Actual huge pages count is 4096 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:619 (2): SPDK autodetecting core list: --lcores (0-3) 2026-02-24T16:42:45.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:45] INFO server.py:633 (2): Starting /usr/local/bin/nvmf_tgt --wait-for-rpc -u -r /var/tmp/spdk.sock --lcores (0-3) 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: pgmap v91: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:45.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: Health check failed: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway. (NVMEOF_SINGLE_GATEWAY) 2026-02-24T16:42:45.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:45.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:45 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2026-02-24T16:42:45.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: pgmap v91: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:45.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:45.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: Health check failed: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway. (NVMEOF_SINGLE_GATEWAY) 2026-02-24T16:42:45.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:45.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:45 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: Health check cleared: NVMEOF_SINGLE_GATEWAY (was: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway.) 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: Cluster is now healthy 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:46.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:46 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: Health check cleared: NVMEOF_SINGLE_GATEWAY (was: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway.) 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: Cluster is now healthy 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:46.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:46 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:46.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2026-02-24T16:42:46.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:46.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "refresh": true, "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-24T16:42:46.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: Health check cleared: NVMEOF_SINGLE_GATEWAY (was: 1 group(s) have only 1 nvmeof gateway; HA is not possible with single gateway.) 2026-02-24T16:42:46.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: Cluster is now healthy 2026-02-24T16:42:46.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:46.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:253 (2): Gateway client.nvmeof.nvmeof.b group id=1 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:268 (2): Stopping the MonitorGroup server... 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:271 (2): The MonitorGroup gRPC server has stopped... 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:576 (2): SPDK Target Path: /usr/local/bin/nvmf_tgt 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:593 (2): SPDK Socket: /var/tmp/spdk.sock 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:612 (2): SPDK will use huge pages, probing... 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:1022 (2): Requested huge pages count is 2048 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:1042 (2): Actual huge pages count is 4096 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:619 (2): SPDK autodetecting core list: --lcores (0-3) 2026-02-24T16:42:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:46] INFO server.py:633 (2): Starting /usr/local/bin/nvmf_tgt --wait-for-rpc -u -r /var/tmp/spdk.sock --lcores (0-3) 2026-02-24T16:42:47.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-mon[43553]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:47.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-mon[43553]: pgmap v92: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:47.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:47.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:47.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-mon[39453]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:47.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-mon[39453]: pgmap v92: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:47.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:47.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:656 (2): SPDK process id: 76 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:659 (2): Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0, log level: WARNING 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: INFO: Setting log level to WARNING 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO client.py:121 (2): Setting log level to WARNING 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.678828] Starting SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2 / DPDK 25.07.0 initialization... 2026-02-24T16:42:47.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.678876] [ DPDK EAL parameters: nvmf --no-shconf --lcores=(0-3) --no-pci --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid76 ] 2026-02-24T16:42:47.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:47 trial153 ceph-mon[44017]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:47.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:47 trial153 ceph-mon[44017]: pgmap v92: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:47.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:47 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:47.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:47 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:253 (2): Gateway client.nvmeof.nvmeof.c group id=2 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:268 (2): Stopping the MonitorGroup server... 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:271 (2): The MonitorGroup gRPC server has stopped... 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:576 (2): SPDK Target Path: /usr/local/bin/nvmf_tgt 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:593 (2): SPDK Socket: /var/tmp/spdk.sock 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:612 (2): SPDK will use huge pages, probing... 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:1022 (2): Requested huge pages count is 2048 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:1042 (2): Actual huge pages count is 4096 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:619 (2): SPDK autodetecting core list: --lcores (0-3) 2026-02-24T16:42:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:47] INFO server.py:633 (2): Starting /usr/local/bin/nvmf_tgt --wait-for-rpc -u -r /var/tmp/spdk.sock --lcores (0-3) 2026-02-24T16:42:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.794250] app.c: 964:spdk_app_start: *NOTICE*: Total cores available: 4 2026-02-24T16:42:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.819369] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 1 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.819378] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 2 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.819387] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 3 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.819388] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 0 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:886 (2): Set RBD CRC32C usage to: True 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:906 (2): DSA acceleration module scanning is disabled 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:47.954971] tcp.c: 822:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** 2026-02-24T16:42:48.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:721 (2): Started SPDK with version "SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2" 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:473 (2): Discovery service process id: 81 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO server.py:461 (81): Starting ceph nvmeof discovery service 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO cephutils.py:186 (2): Connected to Ceph with version "20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev - RelWithDebInfo)" 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO state.py:894 (81): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO discovery.py:344 (81): log pages info from omap: nvmeof.mygroup0.state 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO discovery.py:369 (81): discovery addr: 10.20.193.151 port: 8009 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO cephutils.py:189 (2): Cluster ID is 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO cluster.py:196 (2): NVMeoF cluster pool size: 32 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO rebalance.py:37 (2): Starting rebalance thread: period: 7, max number ns to move: 8 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO state.py:2073 (81): Initialization is over (discovery-trial151). 2026-02-24T16:42:48.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO grpc.py:987 (2): Will enable gateway's IO statistics 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:47] INFO grpc.py:7420 (2): Received request to set gateway's IO statistics mode, enabled: True 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:48.002225] tcp.c:3713:nvmf_tcp_enable_stats: *NOTICE*: Allocated pool for IO statistics pool size 65536 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:42:48.002257] tcp.c:3722:nvmf_tcp_enable_stats: *NOTICE*: IO Statistics enable = 1 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:48] INFO state.py:2073 (2): Initialization is over (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:48] INFO cephutils.py:308 (2): Registered nvmeof.a to service_map! 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:48] INFO server.py:278 (2): Prometheus exporter endpoint is enabled 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:48] INFO server.py:283 (2): Delaying Prometheus exporter startup by 240 seconds... 2026-02-24T16:42:48.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:42:48] INFO server.py:290 (2): Prometheus exporter startup scheduled in background 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1887623997' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2451215769' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"} : dispatch 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"} : dispatch 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"} : dispatch 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"} : dispatch 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:48 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1887623997' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2451215769' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"} : dispatch 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"} : dispatch 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"} : dispatch 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"} : dispatch 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.737 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:48.769 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2026-02-24T16:42:48.769 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1887623997' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:48.769 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.769 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2451215769' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"} : dispatch 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.770 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"} : dispatch 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"} : dispatch 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.771 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"} : dispatch 2026-02-24T16:42:48.772 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.772 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:48.772 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:42:48.772 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:656 (2): SPDK process id: 127 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:659 (2): Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0, log level: WARNING 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: INFO: Setting log level to WARNING 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO client.py:121 (2): Setting log level to WARNING 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.560367] Starting SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2 / DPDK 25.07.0 initialization... 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.560420] [ DPDK EAL parameters: nvmf --no-shconf --lcores=(0-3) --no-pci --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid127 ] 2026-02-24T16:42:48.773 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.676662] app.c: 964:spdk_app_start: *NOTICE*: Total cores available: 4 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.709992] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 1 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.710004] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 2 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.710014] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 3 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.710015] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 0 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:886 (2): Set RBD CRC32C usage to: True 2026-02-24T16:42:48.774 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:906 (2): DSA acceleration module scanning is disabled 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.829482] tcp.c: 822:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:721 (2): Started SPDK with version "SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2" 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:473 (2): Discovery service process id: 132 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:461 (132): Starting ceph nvmeof discovery service 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO cephutils.py:186 (2): Connected to Ceph with version "20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev - RelWithDebInfo)" 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO state.py:894 (132): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO discovery.py:344 (132): log pages info from omap: nvmeof.mygroup0.state 2026-02-24T16:42:49.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO discovery.py:369 (132): discovery addr: 10.20.193.153 port: 8009 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO cephutils.py:189 (2): Cluster ID is 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO cluster.py:196 (2): NVMeoF cluster pool size: 32 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO rebalance.py:37 (2): Starting rebalance thread: period: 7, max number ns to move: 8 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO grpc.py:987 (2): Will enable gateway's IO statistics 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO state.py:2073 (132): Initialization is over (discovery-trial153). 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO grpc.py:7420 (2): Received request to set gateway's IO statistics mode, enabled: True 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.872715] tcp.c:3713:nvmf_tcp_enable_stats: *NOTICE*: Allocated pool for IO statistics pool size 65536 2026-02-24T16:42:49.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:42:48.872748] tcp.c:3722:nvmf_tcp_enable_stats: *NOTICE*: IO Statistics enable = 1 2026-02-24T16:42:49.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO state.py:2073 (2): Initialization is over (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:42:49.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO cephutils.py:308 (2): Registered nvmeof.b to service_map! 2026-02-24T16:42:49.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:278 (2): Prometheus exporter endpoint is enabled 2026-02-24T16:42:49.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:283 (2): Delaying Prometheus exporter startup by 240 seconds... 2026-02-24T16:42:49.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:42:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:42:48] INFO server.py:290 (2): Prometheus exporter startup scheduled in background 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:253 (2): Gateway client.nvmeof.nvmeof.d group id=3 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:268 (2): Stopping the MonitorGroup server... 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:271 (2): The MonitorGroup gRPC server has stopped... 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:576 (2): SPDK Target Path: /usr/local/bin/nvmf_tgt 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:593 (2): SPDK Socket: /var/tmp/spdk.sock 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:612 (2): SPDK will use huge pages, probing... 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:1022 (2): Requested huge pages count is 2048 2026-02-24T16:42:49.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:1042 (2): Actual huge pages count is 4096 2026-02-24T16:42:49.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:619 (2): SPDK autodetecting core list: --lcores (0-3) 2026-02-24T16:42:49.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:49] INFO server.py:633 (2): Starting /usr/local/bin/nvmf_tgt --wait-for-rpc -u -r /var/tmp/spdk.sock --lcores (0-3) 2026-02-24T16:42:49.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: Adding NVMeoF gateway 10.20.193.151:5500 to Dashboard 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: Adding NVMeoF gateway 10.20.193.153:5500 to Dashboard 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: Adding NVMeoF gateway 10.20.193.183:5500 to Dashboard 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: Adding NVMeoF gateway 10.20.193.191:5500 to Dashboard 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"}]: dispatch 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"}]: dispatch 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"}]: dispatch 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"}]: dispatch 2026-02-24T16:42:49.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: pgmap v93: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/250587715' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:49.816 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/972154732' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: Adding NVMeoF gateway 10.20.193.151:5500 to Dashboard 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: Adding NVMeoF gateway 10.20.193.153:5500 to Dashboard 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: Adding NVMeoF gateway 10.20.193.183:5500 to Dashboard 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: Adding NVMeoF gateway 10.20.193.191:5500 to Dashboard 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"}]: dispatch 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"}]: dispatch 2026-02-24T16:42:49.876 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"}]: dispatch 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"}]: dispatch 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: pgmap v93: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.877 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/250587715' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:49.878 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/972154732' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: Adding NVMeoF gateway 10.20.193.151:5500 to Dashboard 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: Adding NVMeoF gateway 10.20.193.153:5500 to Dashboard 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: Adding NVMeoF gateway 10.20.193.183:5500 to Dashboard 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: Adding NVMeoF gateway 10.20.193.191:5500 to Dashboard 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.a"}]: dispatch 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.b"}]: dispatch 2026-02-24T16:42:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.c"}]: dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0", "group": "mygroup0", "daemon_name": "nvmeof.nvmeof.d"}]: dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: pgmap v93: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/250587715' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/972154732' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:49] INFO server.py:656 (2): SPDK process id: 100 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:49] INFO server.py:659 (2): Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0, log level: WARNING 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: INFO: Setting log level to WARNING 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:49] INFO client.py:121 (2): Setting log level to WARNING 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:49.891341] Starting SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2 / DPDK 25.07.0 initialization... 2026-02-24T16:42:50.158 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:49.891390] [ DPDK EAL parameters: nvmf --no-shconf --lcores=(0-3) --no-pci --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid100 ] 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.008871] app.c: 964:spdk_app_start: *NOTICE*: Total cores available: 4 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.039258] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 1 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.039267] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 2 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.039275] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 3 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.039276] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 0 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:886 (2): Set RBD CRC32C usage to: True 2026-02-24T16:42:50.159 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:906 (2): DSA acceleration module scanning is disabled 2026-02-24T16:42:50.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.163189] tcp.c: 822:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** 2026-02-24T16:42:50.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:721 (2): Started SPDK with version "SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2" 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:473 (2): Discovery service process id: 105 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:461 (105): Starting ceph nvmeof discovery service 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO cephutils.py:186 (2): Connected to Ceph with version "20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev - RelWithDebInfo)" 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO state.py:894 (105): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO discovery.py:344 (105): log pages info from omap: nvmeof.mygroup0.state 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO discovery.py:369 (105): discovery addr: 10.20.193.183 port: 8009 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO cephutils.py:189 (2): Cluster ID is 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO cluster.py:196 (2): NVMeoF cluster pool size: 32 2026-02-24T16:42:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO rebalance.py:37 (2): Starting rebalance thread: period: 7, max number ns to move: 8 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO grpc.py:987 (2): Will enable gateway's IO statistics 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO state.py:2073 (105): Initialization is over (discovery-trial183). 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO grpc.py:7420 (2): Received request to set gateway's IO statistics mode, enabled: True 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.206419] tcp.c:3713:nvmf_tcp_enable_stats: *NOTICE*: Allocated pool for IO statistics pool size 65536 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:42:50.206443] tcp.c:3722:nvmf_tcp_enable_stats: *NOTICE*: IO Statistics enable = 1 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO state.py:2073 (2): Initialization is over (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO cephutils.py:308 (2): Registered nvmeof.c to service_map! 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:278 (2): Prometheus exporter endpoint is enabled 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:283 (2): Delaying Prometheus exporter startup by 240 seconds... 2026-02-24T16:42:50.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:42:50] INFO server.py:290 (2): Prometheus exporter startup scheduled in background 2026-02-24T16:42:51.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1552193555' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:51.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3693652473' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:51.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:50 trial183 ceph-mon[43553]: pgmap v94: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1552193555' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3693652473' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:50 trial151 ceph-mon[39453]: pgmap v94: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:51.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1552193555' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:51.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3693652473' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:51.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:50 trial153 ceph-mon[44017]: pgmap v94: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-24T16:42:51.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:656 (2): SPDK process id: 121 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:659 (2): Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0, log level: WARNING 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: INFO: Setting log level to WARNING 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO client.py:121 (2): Setting log level to WARNING 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.095652] Starting SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2 / DPDK 25.07.0 initialization... 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.095795] [ DPDK EAL parameters: nvmf --no-shconf --lcores=(0-3) --no-pci --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid121 ] 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.213056] app.c: 964:spdk_app_start: *NOTICE*: Total cores available: 4 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.238714] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 1 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.238723] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 2 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.238732] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 3 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.238733] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 0 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:886 (2): Set RBD CRC32C usage to: True 2026-02-24T16:42:51.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:906 (2): DSA acceleration module scanning is disabled 2026-02-24T16:42:51.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.354705] tcp.c: 822:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:721 (2): Started SPDK with version "SPDK v25.09 git sha1 77d121fa4e83eb534daaccf3e377c2dfc1b8b1d2" 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:473 (2): Discovery service process id: 126 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:461 (126): Starting ceph nvmeof discovery service 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO cephutils.py:186 (2): Connected to Ceph with version "20.3.0-5507-g618c318a (618c318a7a0b287ab5aeff207b7bf1d73ef45044) tentacle (dev - RelWithDebInfo)" 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO state.py:894 (126): nvmeof.mygroup0.state OMAP object already exists. 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO discovery.py:344 (126): log pages info from omap: nvmeof.mygroup0.state 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO discovery.py:369 (126): discovery addr: 10.20.193.191 port: 8009 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO state.py:2073 (126): Initialization is over (discovery-trial191). 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO cephutils.py:189 (2): Cluster ID is 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO cluster.py:196 (2): NVMeoF cluster pool size: 32 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO rebalance.py:37 (2): Starting rebalance thread: period: 7, max number ns to move: 8 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO grpc.py:987 (2): Will enable gateway's IO statistics 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO grpc.py:7420 (2): Received request to set gateway's IO statistics mode, enabled: True 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.401928] tcp.c:3713:nvmf_tcp_enable_stats: *NOTICE*: Allocated pool for IO statistics pool size 65536 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:42:51.401956] tcp.c:3722:nvmf_tcp_enable_stats: *NOTICE*: IO Statistics enable = 1 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO state.py:2073 (2): Initialization is over (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO cephutils.py:308 (2): Registered nvmeof.d to service_map! 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:278 (2): Prometheus exporter endpoint is enabled 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:283 (2): Delaying Prometheus exporter startup by 240 seconds... 2026-02-24T16:42:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:42:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:42:51] INFO server.py:290 (2): Prometheus exporter startup scheduled in background 2026-02-24T16:42:52.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1997264331' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3967417076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:52.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1997264331' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:52.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3967417076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:52.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1997264331' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2026-02-24T16:42:52.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3967417076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:53.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:52 trial183 ceph-mon[43553]: pgmap v95: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:53.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:52 trial151 ceph-mon[39453]: pgmap v95: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:53.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:52 trial153 ceph-mon[44017]: pgmap v95: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:55 trial183 ceph-mon[43553]: pgmap v96: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4050740896' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:55 trial151 ceph-mon[39453]: pgmap v96: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4050740896' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:55 trial153 ceph-mon[44017]: pgmap v96: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4050740896' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:56.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:56 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4110564765' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:56.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:56 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4110564765' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:57.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:56 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4110564765' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:57.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:57 trial183 ceph-mon[43553]: pgmap v97: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:57.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1413619034' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:57.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:57 trial151 ceph-mon[39453]: pgmap v97: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:57.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1413619034' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:58.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:57 trial153 ceph-mon[44017]: pgmap v97: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2026-02-24T16:42:58.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1413619034' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2612294166' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:58.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2612294166' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:59.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2612294166' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:42:59.885 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:42:59 trial151 ceph-mon[39453]: pgmap v98: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:42:59.964 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:42:59 trial153 ceph-mon[44017]: pgmap v98: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:42:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:42:59 trial183 ceph-mon[43553]: pgmap v98: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:01 trial183 ceph-mon[43553]: pgmap v99: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:01.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:01 trial151 ceph-mon[39453]: pgmap v99: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:01 trial153 ceph-mon[44017]: pgmap v99: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:02.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/378303468' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/378303468' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/378303468' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:03.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:03 trial183 ceph-mon[43553]: pgmap v100: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:03.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:03 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1071116651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:03.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:03 trial151 ceph-mon[39453]: pgmap v100: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:03.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:03 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1071116651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:03 trial153 ceph-mon[44017]: pgmap v100: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:03 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1071116651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:04.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:04 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3203848924' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:04.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:04 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3203848924' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:05.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:04 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3203848924' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:05.544 INFO:tasks.workunit.client.0.trial051.stderr:Note: switching to '618c318a7a0b287ab5aeff207b7bf1d73ef45044'. 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr:state without impacting any branches by switching back to a branch. 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr: git switch -c 2026-02-24T16:43:05.545 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr:Or undo this operation with: 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr: git switch - 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:43:05.546 INFO:tasks.workunit.client.0.trial051.stderr:HEAD is now at 618c318a7a0 Merge branch 'stretched-cluster' of https://github.com/leonidc/ceph into wip-yuri8-testing-2026-02-23-2128 2026-02-24T16:43:05.552 DEBUG:teuthology.orchestra.run.trial051:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-24T16:43:05.609 INFO:tasks.workunit.client.0.trial051.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-24T16:43:05.611 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T16:43:05.611 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-24T16:43:05.708 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-24T16:43:05.739 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-24T16:43:05.762 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T16:43:05.764 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T16:43:05.764 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-24T16:43:05.789 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T16:43:05.793 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:43:05.793 DEBUG:teuthology.orchestra.run.trial051:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-24T16:43:05.849 INFO:tasks.workunit:Running workunits matching nvmeof/setup_subsystem.sh on client.0... 2026-02-24T16:43:05.851 INFO:tasks.workunit:Running workunit nvmeof/setup_subsystem.sh... 2026-02-24T16:43:05.851 DEBUG:teuthology.orchestra.run.trial051:workunit test nvmeof/setup_subsystem.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_IMAGE_PREFIX=myimage RBD_POOL=mypool timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/setup_subsystem.sh 2026-02-24T16:43:05.906 INFO:tasks.workunit.client.0.trial051.stderr:+ source /etc/ceph/nvmeof.env 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_GATEWAY_NAMES=trial151,trial153,trial183,trial191 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=10.20.193.151 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:devel 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_PORT=4420 2026-02-24T16:43:05.907 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SRPORT=5500 2026-02-24T16:43:05.908 INFO:tasks.workunit.client.0.trial051.stderr:+ RBD_POOL=mypool 2026-02-24T16:43:05.908 INFO:tasks.workunit.client.0.trial051.stderr:+ RBD_IMAGE_PREFIX=myimage 2026-02-24T16:43:05.908 INFO:tasks.workunit.client.0.trial051.stderr:++ hostname 2026-02-24T16:43:05.908 INFO:tasks.workunit.client.0.trial051.stderr:+ HOSTNAME=trial051 2026-02-24T16:43:05.908 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman images 2026-02-24T16:43:05.929 INFO:tasks.workunit.client.0.trial051.stdout:REPOSITORY TAG IMAGE ID CREATED SIZE 2026-02-24T16:43:05.929 INFO:tasks.workunit.client.0.trial051.stdout:quay.ceph.io/ceph-ci/ceph 618c318a7a0b287ab5aeff207b7bf1d73ef45044 c2b2dcd214a8 9 hours ago 1.6 GB 2026-02-24T16:43:05.930 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman ps 2026-02-24T16:43:05.950 INFO:tasks.workunit.client.0.trial051.stdout:CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2026-02-24T16:43:05.952 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json subsystem list 2026-02-24T16:43:05.971 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:05Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:05.972 INFO:tasks.workunit.client.0.trial051.stderr:Trying to pull quay.io/ceph/nvmeof-cli:devel... 2026-02-24T16:43:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:05 trial183 ceph-mon[43553]: pgmap v101: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1553107993' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:05 trial151 ceph-mon[39453]: pgmap v101: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1553107993' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:05 trial153 ceph-mon[44017]: pgmap v101: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1553107993' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:06.330 INFO:tasks.workunit.client.0.trial051.stderr:Getting image source signatures 2026-02-24T16:43:06.331 INFO:tasks.workunit.client.0.trial051.stderr:Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2026-02-24T16:43:06.331 INFO:tasks.workunit.client.0.trial051.stderr:Copying blob sha256:c9bc245d94649254d6eda62be059895c888a58c8efb125021ae7f0fef67431f1 2026-02-24T16:43:06.331 INFO:tasks.workunit.client.0.trial051.stderr:Copying blob sha256:c13c0b6a7c5c605d1e299619b7f0853147f52149077394f70a66efdcf16fc6fb 2026-02-24T16:43:06.331 INFO:tasks.workunit.client.0.trial051.stderr:Copying blob sha256:271b82640408a5fd9fe60b060a52f3a26008cf3fd1a7e5ab3e544b00287aa6d4 2026-02-24T16:43:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:07 trial183 ceph-mon[43553]: pgmap v102: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:07.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:07 trial151 ceph-mon[39453]: pgmap v102: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:08.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:07 trial153 ceph-mon[44017]: pgmap v102: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:09.827 INFO:tasks.workunit.client.0.trial051.stderr:Copying config sha256:c788be2c0094e4bcee2bb36f7aaa7cc9bb9c15925ad52cf69bb46c41a4a18334 2026-02-24T16:43:09.828 INFO:tasks.workunit.client.0.trial051.stderr:Writing manifest to image destination 2026-02-24T16:43:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:09 trial183 ceph-mon[43553]: pgmap v103: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2896489239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:09 trial151 ceph-mon[39453]: pgmap v103: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2896489239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:09 trial153 ceph-mon[44017]: pgmap v103: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:43:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2896489239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:10.305 INFO:tasks.workunit.client.0.trial051.stdout:{ 2026-02-24T16:43:10.306 INFO:tasks.workunit.client.0.trial051.stdout: "error_message": "Success", 2026-02-24T16:43:10.306 INFO:tasks.workunit.client.0.trial051.stdout: "status": 0, 2026-02-24T16:43:10.306 INFO:tasks.workunit.client.0.trial051.stdout: "subsystems": [] 2026-02-24T16:43:10.306 INFO:tasks.workunit.client.0.trial051.stdout:} 2026-02-24T16:43:10.326 INFO:tasks.workunit.client.0.trial051.stderr:+ IFS=, 2026-02-24T16:43:10.326 INFO:tasks.workunit.client.0.trial051.stderr:+ read -ra gateway_ips 2026-02-24T16:43:10.326 INFO:tasks.workunit.client.0.trial051.stderr:+ IFS=, 2026-02-24T16:43:10.326 INFO:tasks.workunit.client.0.trial051.stderr:+ read -ra gateway_names 2026-02-24T16:43:10.327 INFO:tasks.workunit.client.0.trial051.stderr:++ echo 10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:43:10.327 INFO:tasks.workunit.client.0.trial051.stderr:++ tr -cd , 2026-02-24T16:43:10.327 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -c 2026-02-24T16:43:10.328 INFO:tasks.workunit.client.0.trial051.stderr:+ gateways_count=4 2026-02-24T16:43:10.328 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Starting subsystem setup...' 2026-02-24T16:43:10.329 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Starting subsystem setup... 2026-02-24T16:43:10.329 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:10.330 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:10.330 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:10.330 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode1 --no-group-append 2026-02-24T16:43:10.367 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:10Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:10.625 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:10] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.51:49546 2026-02-24T16:43:10.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:10 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1788241455' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:10.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:10 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1788241455' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:11.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:10 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1788241455' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:11.353 INFO:tasks.workunit.client.0.trial051.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:11.373 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:11.373 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:11.373 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode2 --no-group-append 2026-02-24T16:43:11.411 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:11Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:11.625 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 0, no group append: True, network mask: [], secure listeners: False, context: , client address: IPv4 10.20.193.51:54802 2026-02-24T16:43:11.625 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:11.625 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1940 (2): No serial number specified for nqn.2016-06.io.spdk:cnode1, will use Ceph96885192365298 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1946 (2): Subsystem NQN will not be changed 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:1053 (2): omap_key generated: subsystem_nqn.2016-06.io.spdk:cnode1, OMAP version: 2 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.626 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.660 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.660 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.936 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-mon[39453]: pgmap v104: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:11.936 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/378004981' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:11.936 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:1684 (81): Start update from 1 to 2 (discovery-trial151). 2026-02-24T16:43:11.936 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:2068 (81): Update complete (1 -> 2) (discovery-trial151). 2026-02-24T16:43:11.936 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 0, no group append: True, network mask: [], secure listeners: False, context: , client address: IPv4 10.20.193.51:43514 2026-02-24T16:43:11.946 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-mon[44017]: pgmap v104: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:11.946 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/378004981' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:11.946 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] INFO state.py:1684 (132): Start update from 1 to 2 (discovery-trial153). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] INFO state.py:2068 (132): Update complete (1 -> 2) (discovery-trial153). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] INFO state.py:1684 (2): Start update from 1 to 2 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] INFO state.py:2068 (2): Update complete (1 -> 2) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] INFO state.py:1684 (126): Start update from 1 to 2 (discovery-trial191). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] INFO state.py:2068 (126): Update complete (1 -> 2) (discovery-trial191). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] INFO state.py:1684 (2): Start update from 1 to 2 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] INFO state.py:2068 (2): Update complete (1 -> 2) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.947 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-mon[43553]: pgmap v104: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:43:11.947 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/378004981' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:11.947 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] INFO state.py:1684 (105): Start update from 1 to 2 (discovery-trial183). 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] INFO state.py:1684 (2): Start update from 1 to 2 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] INFO state.py:2068 (105): Update complete (1 -> 2) (discovery-trial183). 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] INFO state.py:2068 (2): Update complete (1 -> 2) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.948 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:11.948 INFO:tasks.workunit.client.0.trial051.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:11.969 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:11.969 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:11.969 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode3 --no-group-append 2026-02-24T16:43:12.010 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:12Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1940 (2): No serial number specified for nqn.2016-06.io.spdk:cnode2, will use Ceph54708660634751 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO grpc.py:1946 (2): Subsystem NQN will not be changed 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] INFO state.py:1053 (2): omap_key generated: subsystem_nqn.2016-06.io.spdk:cnode2, OMAP version: 3 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:11] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.509 INFO:tasks.workunit.client.0.trial051.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:12.530 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:43:12.531 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.151 2026-02-24T16:43:12.531 INFO:tasks.workunit.client.0.trial051.stderr:+ name=trial151 2026-02-24T16:43:12.531 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:12.532 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:12.532 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:12.532 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.151 and name trial151 2026-02-24T16:43:12.532 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.151 and name trial151' 2026-02-24T16:43:12.533 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name trial151 --traddr 10.20.193.151 --trsvcid 4420 2026-02-24T16:43:12.568 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:12Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:12.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] INFO state.py:1684 (105): Start update from 2 to 3 (discovery-trial183). 2026-02-24T16:43:12.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] INFO state.py:2068 (105): Update complete (2 -> 3) (discovery-trial183). 2026-02-24T16:43:12.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] INFO state.py:1684 (2): Start update from 2 to 3 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:12.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:12.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] INFO state.py:2068 (2): Update complete (2 -> 3) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:12.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:12] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO state.py:1684 (81): Start update from 2 to 3 (discovery-trial151). 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO state.py:2068 (81): Update complete (2 -> 3) (discovery-trial151). 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 0, no group append: True, network mask: [], secure listeners: False, context: , client address: IPv4 10.20.193.51:58998 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO grpc.py:1940 (2): No serial number specified for nqn.2016-06.io.spdk:cnode3, will use Ceph87132565256667 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO grpc.py:1946 (2): Subsystem NQN will not be changed 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] INFO state.py:1053 (2): omap_key generated: subsystem_nqn.2016-06.io.spdk:cnode3, OMAP version: 4 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] INFO state.py:1684 (2): Start update from 2 to 3 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] INFO state.py:1684 (132): Start update from 2 to 3 (discovery-trial153). 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] INFO state.py:2068 (132): Update complete (2 -> 3) (discovery-trial153). 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] INFO state.py:2068 (2): Update complete (2 -> 3) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] INFO state.py:1684 (126): Start update from 2 to 3 (discovery-trial191). 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] INFO state.py:2068 (126): Update complete (2 -> 3) (discovery-trial191). 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] INFO state.py:1684 (2): Start update from 2 to 3 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] INFO state.py:2068 (2): Update complete (2 -> 3) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:12.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.093 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 10.20.193.151:4420: Successful 2026-02-24T16:43:13.112 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:13.112 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:13.112 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.151 and name trial151' 2026-02-24T16:43:13.112 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.151 and name trial151 2026-02-24T16:43:13.112 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name trial151 --traddr 10.20.193.151 --trsvcid 4420 2026-02-24T16:43:13.155 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:13Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:13.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:1684 (132): Start update from 3 to 4 (discovery-trial153). 2026-02-24T16:43:13.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:2068 (132): Update complete (3 -> 4) (discovery-trial153). 2026-02-24T16:43:13.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 3 to 4 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:13.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:13.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (3 -> 4) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:13.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:1684 (126): Start update from 3 to 4 (discovery-trial191). 2026-02-24T16:43:13.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 3 to 4 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:13.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:2068 (126): Update complete (3 -> 4) (discovery-trial191). 2026-02-24T16:43:13.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:13.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (3 -> 4) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:13.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 3 to 4 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:13.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:1684 (105): Start update from 3 to 4 (discovery-trial183). 2026-02-24T16:43:13.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO grpc.py:1834 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 512, no group append: True, network mask: [], secure listeners: False, context: None 2026-02-24T16:43:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:2068 (105): Update complete (3 -> 4) (discovery-trial183). 2026-02-24T16:43:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (3 -> 4) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:1684 (81): Start update from 3 to 4 (discovery-trial151). 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:2068 (81): Update complete (3 -> 4) (discovery-trial151). 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.151:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:38210 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:13.079746] tcp.c:1165:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.20.193.151 port 4420 *** 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode1_trial151_TCP_10.20.193.151_4420, OMAP version: 5 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.645 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 10.20.193.151:4420: Successful 2026-02-24T16:43:13.665 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:13.666 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:13.666 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.151 and name trial151 2026-02-24T16:43:13.667 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.151 and name trial151' 2026-02-24T16:43:13.667 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name trial151 --traddr 10.20.193.151 --trsvcid 4420 2026-02-24T16:43:13.710 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:13Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-mon[43553]: pgmap v105: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 11 KiB/s rd, 341 B/s wr, 16 op/s 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 4 to 5 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial151 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (4 -> 5) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:1684 (105): Start update from 4 to 5 (discovery-trial183). 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] INFO state.py:2068 (105): Update complete (4 -> 5) (discovery-trial183). 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:13] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-mon[39453]: pgmap v105: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 11 KiB/s rd, 341 B/s wr, 16 op/s 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:1684 (81): Start update from 4 to 5 (discovery-trial151). 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:2068 (81): Update complete (4 -> 5) (discovery-trial151). 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.151:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:50036 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode2_trial151_TCP_10.20.193.151_4420, OMAP version: 6 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-mon[44017]: pgmap v105: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 11 KiB/s rd, 341 B/s wr, 16 op/s 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 4 to 5 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:1684 (132): Start update from 4 to 5 (discovery-trial153). 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:2068 (132): Update complete (4 -> 5) (discovery-trial153). 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial151 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (4 -> 5) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:13] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:1684 (126): Start update from 4 to 5 (discovery-trial191). 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:2068 (126): Update complete (4 -> 5) (discovery-trial191). 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:1684 (2): Start update from 4 to 5 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial151 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] INFO state.py:2068 (2): Update complete (4 -> 5) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.189 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 10.20.193.151:4420: Successful 2026-02-24T16:43:14.209 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:43:14.209 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.153 2026-02-24T16:43:14.209 INFO:tasks.workunit.client.0.trial051.stderr:+ name=trial153 2026-02-24T16:43:14.209 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:14.211 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:14.211 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:14.211 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.153 and name trial153' 2026-02-24T16:43:14.212 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.153 and name trial153 2026-02-24T16:43:14.212 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.153 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name trial153 --traddr 10.20.193.153 --trsvcid 4420 2026-02-24T16:43:14.249 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 5 to 6 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial151 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (5 -> 6) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:1684 (105): Start update from 5 to 6 (discovery-trial183). 2026-02-24T16:43:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:2068 (105): Update complete (5 -> 6) (discovery-trial183). 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:1684 (81): Start update from 5 to 6 (discovery-trial151). 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:2068 (81): Update complete (5 -> 6) (discovery-trial151). 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.151:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:49056 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode3_trial151_TCP_10.20.193.151_4420, OMAP version: 7 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 5 to 6 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial151 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (5 -> 6) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:1684 (132): Start update from 5 to 6 (discovery-trial153). 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:2068 (132): Update complete (5 -> 6) (discovery-trial153). 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:1684 (126): Start update from 5 to 6 (discovery-trial191). 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:2068 (126): Update complete (5 -> 6) (discovery-trial191). 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 5 to 6 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial151 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (5 -> 6) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:14.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.759 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 10.20.193.153:4420: Successful 2026-02-24T16:43:14.782 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:14.782 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:14.782 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.153 and name trial153 2026-02-24T16:43:14.783 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.153 and name trial153' 2026-02-24T16:43:14.783 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.153 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name trial153 --traddr 10.20.193.153 --trsvcid 4420 2026-02-24T16:43:14.821 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 6 to 7 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial151 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (6 -> 7) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:1684 (105): Start update from 6 to 7 (discovery-trial183). 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] INFO state.py:2068 (105): Update complete (6 -> 7) (discovery-trial183). 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:14] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:1684 (81): Start update from 6 to 7 (discovery-trial151). 2026-02-24T16:43:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] INFO state.py:2068 (81): Update complete (6 -> 7) (discovery-trial151). 2026-02-24T16:43:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:14] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 6 to 7 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:15.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial151 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (6 -> 7) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:1684 (132): Start update from 6 to 7 (discovery-trial153). 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:2068 (132): Update complete (6 -> 7) (discovery-trial153). 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.153:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:41524 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:14.749527] tcp.c:1165:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.20.193.153 port 4420 *** 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode1_trial153_TCP_10.20.193.153_4420, OMAP version: 8 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:1684 (126): Start update from 6 to 7 (discovery-trial191). 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:2068 (126): Update complete (6 -> 7) (discovery-trial191). 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:1684 (2): Start update from 6 to 7 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO grpc.py:6313 (2): Received request to create trial151 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.151:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial151 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] INFO state.py:2068 (2): Update complete (6 -> 7) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.315 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 10.20.193.153:4420: Successful 2026-02-24T16:43:15.335 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:15.335 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:15.335 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.153 and name trial153' 2026-02-24T16:43:15.335 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.153 and name trial153 2026-02-24T16:43:15.336 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.153 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name trial153 --traddr 10.20.193.153 --trsvcid 4420 2026-02-24T16:43:15.363 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:15Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:1684 (132): Start update from 7 to 8 (discovery-trial153). 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:2068 (132): Update complete (7 -> 8) (discovery-trial153). 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.153:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:50446 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode2_trial153_TCP_10.20.193.153_4420, OMAP version: 9 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:1684 (126): Start update from 7 to 8 (discovery-trial191). 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:2068 (126): Update complete (7 -> 8) (discovery-trial191). 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 7 to 8 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial153 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (7 -> 8) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 7 to 8 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:1684 (81): Start update from 7 to 8 (discovery-trial151). 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial153 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:2068 (81): Update complete (7 -> 8) (discovery-trial151). 2026-02-24T16:43:15.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (7 -> 8) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:1684 (105): Start update from 7 to 8 (discovery-trial183). 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:2068 (105): Update complete (7 -> 8) (discovery-trial183). 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 7 to 8 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial153 2026-02-24T16:43:15.633 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (7 -> 8) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:15.633 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.633 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.855 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 10.20.193.153:4420: Successful 2026-02-24T16:43:15.874 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:43:15.874 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.183 2026-02-24T16:43:15.875 INFO:tasks.workunit.client.0.trial051.stderr:+ name=trial183 2026-02-24T16:43:15.875 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:15.875 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:15.876 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:15.876 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.183 and name trial183' 2026-02-24T16:43:15.876 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.183 and name trial183 2026-02-24T16:43:15.876 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.183 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name trial183 --traddr 10.20.193.183 --trsvcid 4420 2026-02-24T16:43:15.913 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:15Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-mon[43553]: pgmap v106: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:1684 (105): Start update from 8 to 9 (discovery-trial183). 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:2068 (105): Update complete (8 -> 9) (discovery-trial183). 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 8 to 9 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial153 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (8 -> 9) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-mon[39453]: pgmap v106: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:15.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:15.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 1 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 1 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 1 2026-02-24T16:43:15.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: ], , client address: IPv4 10.20.193.151:38252 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 8 to 9 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:1684 (81): Start update from 8 to 9 (discovery-trial151). 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:2068 (81): Update complete (8 -> 9) (discovery-trial151). 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial153 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (8 -> 9) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:15.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:15] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-mon[44017]: pgmap v106: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 2 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:16.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 2 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 2 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: ], , client address: IPv4 10.20.193.153:55094 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:1684 (132): Start update from 8 to 9 (discovery-trial153). 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:2068 (132): Update complete (8 -> 9) (discovery-trial153). 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.153:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:43814 2026-02-24T16:43:16.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:16.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode3_trial153_TCP_10.20.193.153_4420, OMAP version: 10 2026-02-24T16:43:16.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.066 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:1684 (126): Start update from 8 to 9 (discovery-trial191). 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:2068 (126): Update complete (8 -> 9) (discovery-trial191). 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:1684 (2): Start update from 8 to 9 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial153 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] INFO state.py:2068 (2): Update complete (8 -> 9) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.401 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 10.20.193.183:4420: Successful 2026-02-24T16:43:16.420 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:16.420 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:16.420 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.183 and name trial183 2026-02-24T16:43:16.421 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.183 and name trial183' 2026-02-24T16:43:16.421 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.183 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name trial183 --traddr 10.20.193.183 --trsvcid 4420 2026-02-24T16:43:16.450 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:16Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:16.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:1684 (105): Start update from 9 to 10 (discovery-trial183). 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:2068 (105): Update complete (9 -> 10) (discovery-trial183). 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 9 to 10 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial153 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (9 -> 10) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.183:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:58268 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:16.390727] tcp.c:1165:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.20.193.183 port 4420 *** 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode1_trial183_TCP_10.20.193.183_4420, OMAP version: 11 2026-02-24T16:43:16.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 9 to 10 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:16.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:16.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial153 2026-02-24T16:43:16.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (9 -> 10) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:16.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:1684 (81): Start update from 9 to 10 (discovery-trial151). 2026-02-24T16:43:16.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:2068 (81): Update complete (9 -> 10) (discovery-trial151). 2026-02-24T16:43:16.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:1684 (132): Start update from 9 to 10 (discovery-trial153). 2026-02-24T16:43:16.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:2068 (132): Update complete (9 -> 10) (discovery-trial153). 2026-02-24T16:43:16.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:1684 (126): Start update from 9 to 10 (discovery-trial191). 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:2068 (126): Update complete (9 -> 10) (discovery-trial191). 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 9 to 10 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial153 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.153:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial153 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (9 -> 10) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:16.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:16.925 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 10.20.193.183:4420: Successful 2026-02-24T16:43:16.944 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:16.945 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:16.945 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.183 and name trial183' 2026-02-24T16:43:16.945 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.183 and name trial183 2026-02-24T16:43:16.945 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.183 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name trial183 --traddr 10.20.193.183 --trsvcid 4420 2026-02-24T16:43:16.984 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:16Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 10 to 11 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial183 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (10 -> 11) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:1684 (81): Start update from 10 to 11 (discovery-trial151). 2026-02-24T16:43:17.187 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] INFO state.py:2068 (81): Update complete (10 -> 11) (discovery-trial151). 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:16] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:17.188 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: OPTIMIZED 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:17.189 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: ], , client address: IPv4 10.20.193.151:38256 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:1684 (132): Start update from 10 to 11 (discovery-trial153). 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:2068 (132): Update complete (10 -> 11) (discovery-trial153). 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 10 to 11 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial183 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (10 -> 11) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:17.190 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: OPTIMIZED 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:17.191 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: ], , client address: IPv4 10.20.193.153:50804 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:1684 (105): Start update from 10 to 11 (discovery-trial183). 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:2068 (105): Update complete (10 -> 11) (discovery-trial183). 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.183:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:57858 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode2_trial183_TCP_10.20.193.183_4420, OMAP version: 12 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:16] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:1684 (126): Start update from 10 to 11 (discovery-trial191). 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:2068 (126): Update complete (10 -> 11) (discovery-trial191). 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:1684 (2): Start update from 10 to 11 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial183 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] INFO state.py:2068 (2): Update complete (10 -> 11) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO state.py:1684 (2): Start update from 11 to 12 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO state.py:1684 (81): Start update from 11 to 12 (discovery-trial151). 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO state.py:2068 (81): Update complete (11 -> 12) (discovery-trial151). 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial183 2026-02-24T16:43:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:17] INFO state.py:2068 (2): Update complete (11 -> 12) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:17.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO state.py:1684 (132): Start update from 11 to 12 (discovery-trial153). 2026-02-24T16:43:17.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO state.py:2068 (132): Update complete (11 -> 12) (discovery-trial153). 2026-02-24T16:43:17.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO state.py:1684 (2): Start update from 11 to 12 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:17.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial183 2026-02-24T16:43:17.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:17] INFO state.py:2068 (2): Update complete (11 -> 12) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:17.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-mon[43553]: pgmap v107: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:17.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4215288935' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:17.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:17] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.183:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:43674 2026-02-24T16:43:17.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:17] WARNING state.py:640 (2): The OMAP file is locked, will try again in 1.0 seconds 2026-02-24T16:43:17.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:17] INFO state.py:1684 (105): Start update from 11 to 12 (discovery-trial183). 2026-02-24T16:43:17.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:17] INFO state.py:2068 (105): Update complete (11 -> 12) (discovery-trial183). 2026-02-24T16:43:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO state.py:1684 (126): Start update from 11 to 12 (discovery-trial191). 2026-02-24T16:43:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO state.py:2068 (126): Update complete (11 -> 12) (discovery-trial191). 2026-02-24T16:43:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO state.py:1684 (2): Start update from 11 to 12 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial183 2026-02-24T16:43:17.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:17] INFO state.py:2068 (2): Update complete (11 -> 12) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:17.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-mon[39453]: pgmap v107: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:17.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:17 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4215288935' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-mon[44017]: pgmap v107: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 8.1 KiB/s rd, 341 B/s wr, 12 op/s 2026-02-24T16:43:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:17 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4215288935' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.449 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 10.20.193.183:4420: Successful 2026-02-24T16:43:18.472 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:43:18.472 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.191 2026-02-24T16:43:18.472 INFO:tasks.workunit.client.0.trial051.stderr:+ name=trial191 2026-02-24T16:43:18.472 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:18.473 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:18.473 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:18.473 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.191 and name trial191' 2026-02-24T16:43:18.474 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.191 and name trial191 2026-02-24T16:43:18.474 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.191 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name trial191 --traddr 10.20.193.191 --trsvcid 4420 2026-02-24T16:43:18.508 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:18.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2766136835' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2923809365' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1352499989' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:18.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode3_trial183_TCP_10.20.193.183_4420, OMAP version: 13 2026-02-24T16:43:18.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2766136835' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2923809365' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1352499989' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2766136835' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2923809365' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1352499989' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:18.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:18.993 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 10.20.193.191:4420: Successful 2026-02-24T16:43:19.014 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:19.014 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:19.014 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.191 and name trial191' 2026-02-24T16:43:19.014 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.191 and name trial191 2026-02-24T16:43:19.014 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.191 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name trial191 --traddr 10.20.193.191 --trsvcid 4420 2026-02-24T16:43:19.053 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:19.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] INFO state.py:1684 (105): Start update from 12 to 13 (discovery-trial183). 2026-02-24T16:43:19.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] INFO state.py:2068 (105): Update complete (12 -> 13) (discovery-trial183). 2026-02-24T16:43:19.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO state.py:1684 (81): Start update from 12 to 13 (discovery-trial151). 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO state.py:2068 (81): Update complete (12 -> 13) (discovery-trial151). 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO state.py:1684 (2): Start update from 12 to 13 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial183 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] INFO state.py:2068 (2): Update complete (12 -> 13) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:18] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO state.py:1684 (132): Start update from 12 to 13 (discovery-trial153). 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO state.py:2068 (132): Update complete (12 -> 13) (discovery-trial153). 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO state.py:1684 (2): Start update from 12 to 13 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial183 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] INFO state.py:2068 (2): Update complete (12 -> 13) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:1684 (126): Start update from 12 to 13 (discovery-trial191). 2026-02-24T16:43:19.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:2068 (126): Update complete (12 -> 13) (discovery-trial191). 2026-02-24T16:43:19.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:1684 (2): Start update from 12 to 13 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:19.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO grpc.py:6313 (2): Received request to create trial183 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.183:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial191 differs from requested host trial183 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:2068 (2): Update complete (12 -> 13) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.191:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:42246 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:18.986644] tcp.c:1165:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.20.193.191 port 4420 *** 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:18] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode1_trial191_TCP_10.20.193.191_4420, OMAP version: 14 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.544 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 10.20.193.191:4420: Successful 2026-02-24T16:43:19.564 INFO:tasks.workunit.client.0.trial051.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:19.564 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:19.565 INFO:tasks.workunit.client.0.trial051.stdout:Adding gateway listener with IP 10.20.193.191 and name trial191 2026-02-24T16:43:19.565 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Adding gateway listener with IP 10.20.193.191 and name trial191' 2026-02-24T16:43:19.565 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.191 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name trial191 --traddr 10.20.193.191 --trsvcid 4420 2026-02-24T16:43:19.601 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:19.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-mon[44017]: pgmap v108: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 43 KiB/s rd, 1.8 KiB/s wr, 54 op/s 2026-02-24T16:43:19.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO state.py:1684 (132): Start update from 13 to 14 (discovery-trial153). 2026-02-24T16:43:19.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO state.py:2068 (132): Update complete (13 -> 14) (discovery-trial153). 2026-02-24T16:43:19.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO state.py:1684 (2): Start update from 13 to 14 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:19.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial191 2026-02-24T16:43:19.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] INFO state.py:2068 (2): Update complete (13 -> 14) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:19.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] INFO state.py:1684 (126): Start update from 13 to 14 (discovery-trial191). 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] INFO state.py:2068 (126): Update complete (13 -> 14) (discovery-trial191). 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.191:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:58246 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode2_trial191_TCP_10.20.193.191_4420, OMAP version: 15 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-mon[43553]: pgmap v108: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 43 KiB/s rd, 1.8 KiB/s wr, 54 op/s 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO state.py:1684 (105): Start update from 13 to 14 (discovery-trial183). 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO state.py:2068 (105): Update complete (13 -> 14) (discovery-trial183). 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO state.py:1684 (2): Start update from 13 to 14 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial191 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] INFO state.py:2068 (2): Update complete (13 -> 14) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-mon[39453]: pgmap v108: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 43 KiB/s rd, 1.8 KiB/s wr, 54 op/s 2026-02-24T16:43:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO state.py:1684 (2): Start update from 13 to 14 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO state.py:1684 (81): Start update from 13 to 14 (discovery-trial151). 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial191 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO state.py:2068 (81): Update complete (13 -> 14) (discovery-trial151). 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO state.py:2068 (2): Update complete (13 -> 14) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:19] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: INACCESSIBLE 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: INACCESSIBLE 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: states { 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: grp_id: 3 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: state: INACCESSIBLE 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: } 2026-02-24T16:43:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: ], , client address: IPv4 10.20.193.151:38296 2026-02-24T16:43:20.084 INFO:tasks.workunit.client.0.trial051.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 10.20.193.191:4420: Successful 2026-02-24T16:43:20.104 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:20.105 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:20.105 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:20.105 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode1 --host '*' 2026-02-24T16:43:20.136 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:20Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:20.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:1684 (132): Start update from 14 to 15 (discovery-trial153). 2026-02-24T16:43:20.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:2068 (132): Update complete (14 -> 15) (discovery-trial153). 2026-02-24T16:43:20.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 14 to 15 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:20.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:20.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial191 2026-02-24T16:43:20.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (14 -> 15) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:20.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:1684 (126): Start update from 14 to 15 (discovery-trial191). 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:2068 (126): Update complete (14 -> 15) (discovery-trial191). 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.191:4420, secure: False, verify host name: False, context: , client address: IPv4 10.20.193.51:60554 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:1053 (2): omap_key generated: listener_nqn.2016-06.io.spdk:cnode3_trial191_TCP_10.20.193.191_4420, OMAP version: 16 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:1684 (105): Start update from 14 to 15 (discovery-trial183). 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:2068 (105): Update complete (14 -> 15) (discovery-trial183). 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 14 to 15 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial191 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (14 -> 15) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 14 to 15 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:1684 (81): Start update from 14 to 15 (discovery-trial151). 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:2068 (81): Update complete (14 -> 15) (discovery-trial151). 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial191 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (14 -> 15) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:20.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:20.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:1684 (126): Start update from 15 to 16 (discovery-trial191). 2026-02-24T16:43:20.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:20] INFO state.py:2068 (126): Update complete (15 -> 16) (discovery-trial191). 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:1684 (105): Start update from 15 to 16 (discovery-trial183). 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:2068 (105): Update complete (15 -> 16) (discovery-trial183). 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 15 to 16 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial183 differs from requested host trial191 2026-02-24T16:43:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (15 -> 16) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.51:46292 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode1 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] WARNING state.py:688 (2): Local version 15 differs from OMAP file version 16. The file is not current, will reload it and try again 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] WARNING state.py:1642 (2): An update is already running, ignore 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 15 to 16 (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial151 differs from requested host trial191 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (15 -> 16) (gateway-client.nvmeof.nvmeof.a). 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:1684 (81): Start update from 15 to 16 (discovery-trial151). 2026-02-24T16:43:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:20] INFO state.py:2068 (81): Update complete (15 -> 16) (discovery-trial151). 2026-02-24T16:43:21.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:1684 (132): Start update from 15 to 16 (discovery-trial153). 2026-02-24T16:43:21.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:2068 (132): Update complete (15 -> 16) (discovery-trial153). 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:1684 (2): Start update from 15 to 16 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO grpc.py:6313 (2): Received request to create trial191 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 10.20.193.191:4420, secure: False, verify host name: True, context: None 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO grpc.py:6382 (2): Listener not created as gateway's host name trial153 differs from requested host trial191 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO state.py:2068 (2): Update complete (15 -> 16) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:20] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: INACCESSIBLE 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: INACCESSIBLE 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: states { 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: grp_id: 4 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: state: INACCESSIBLE 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: } 2026-02-24T16:43:21.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: ], , client address: IPv4 10.20.193.153:50844 2026-02-24T16:43:21.085 INFO:tasks.workunit.client.0.trial051.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:21.085 INFO:tasks.workunit.client.0.trial051.stdout:Open host access to subsystem nqn.2016-06.io.spdk:cnode1 might be a security breach 2026-02-24T16:43:21.106 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:21.106 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:21.106 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode2 --host '*' 2026-02-24T16:43:21.145 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:21Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:21.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.51:46292 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode1 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO state.py:1053 (2): omap_key generated: host_nqn.2016-06.io.spdk:cnode1_*, OMAP version: 17 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] INFO state.py:1684 (126): Start update from 16 to 17 (discovery-trial191). 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] INFO state.py:2068 (126): Update complete (16 -> 17) (discovery-trial191). 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] INFO state.py:1684 (2): Start update from 16 to 17 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode1 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:21.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:21] INFO state.py:2068 (2): Update complete (16 -> 17) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:21.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-mon[43553]: pgmap v109: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 50 op/s 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] INFO state.py:1684 (2): Start update from 16 to 17 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode1 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] INFO state.py:1684 (105): Start update from 16 to 17 (discovery-trial183). 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] INFO state.py:2068 (105): Update complete (16 -> 17) (discovery-trial183). 2026-02-24T16:43:21.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:21] INFO state.py:2068 (2): Update complete (16 -> 17) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:21.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-mon[39453]: pgmap v109: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 50 op/s 2026-02-24T16:43:21.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.51:59032 2026-02-24T16:43:21.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode2 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:21.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO state.py:1684 (81): Start update from 16 to 17 (discovery-trial151). 2026-02-24T16:43:21.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] INFO state.py:2068 (81): Update complete (16 -> 17) (discovery-trial151). 2026-02-24T16:43:21.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:21] WARNING state.py:640 (2): The OMAP file is locked, will try again in 1.0 seconds 2026-02-24T16:43:22.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-mon[44017]: pgmap v109: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 50 op/s 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] INFO state.py:1684 (2): Start update from 16 to 17 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode1 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] INFO state.py:1684 (132): Start update from 16 to 17 (discovery-trial153). 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] INFO state.py:2068 (132): Update complete (16 -> 17) (discovery-trial153). 2026-02-24T16:43:22.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:21] INFO state.py:2068 (2): Update complete (16 -> 17) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:22.601 INFO:tasks.workunit.client.0.trial051.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:22.601 INFO:tasks.workunit.client.0.trial051.stdout:Open host access to subsystem nqn.2016-06.io.spdk:cnode2 might be a security breach 2026-02-24T16:43:22.622 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:22.622 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:22.622 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode3 --host '*' 2026-02-24T16:43:22.659 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:22Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:22] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:22] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:22] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: states { 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: grp_id: 3 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: state: OPTIMIZED 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: } 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: states { 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: grp_id: 3 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: state: OPTIMIZED 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: } 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: states { 2026-02-24T16:43:22.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: grp_id: 3 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: state: OPTIMIZED 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: } 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:22 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: ], , client address: IPv4 10.20.193.183:33168 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:22 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:22] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:22 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:22] INFO state.py:1053 (2): omap_key generated: host_nqn.2016-06.io.spdk:cnode2_*, OMAP version: 18 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:22 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:22] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:22.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:22 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:22] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:22 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:22] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:22 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:22] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:22 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:22] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:22 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:22] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.125 INFO:tasks.workunit.client.0.trial051.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:23.125 INFO:tasks.workunit.client.0.trial051.stdout:Open host access to subsystem nqn.2016-06.io.spdk:cnode3 might be a security breach 2026-02-24T16:43:23.144 INFO:tasks.workunit.client.0.trial051.stderr:+ image_index=1 2026-02-24T16:43:23.144 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 3 2026-02-24T16:43:23.145 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:23.145 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:43:23.146 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 20 2026-02-24T16:43:23.146 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:23.147 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage1 2026-02-24T16:43:23.147 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage1 --load-balancing-group 2 2026-02-24T16:43:23.193 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:23Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 17 to 18 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode2 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:1684 (105): Start update from 17 to 18 (discovery-trial183). 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:2068 (105): Update complete (17 -> 18) (discovery-trial183). 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (17 -> 18) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:1684 (81): Start update from 17 to 18 (discovery-trial151). 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:2068 (81): Update complete (17 -> 18) (discovery-trial151). 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.51:47538 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode3 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:1053 (2): omap_key generated: host_nqn.2016-06.io.spdk:cnode3_*, OMAP version: 19 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 17 to 18 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode2 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:1684 (132): Start update from 17 to 18 (discovery-trial153). 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:2068 (132): Update complete (17 -> 18) (discovery-trial153). 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (17 -> 18) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:1684 (126): Start update from 17 to 18 (discovery-trial191). 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:2068 (126): Update complete (17 -> 18) (discovery-trial191). 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 17 to 18 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode2 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (17 -> 18) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.850 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:23.878 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:23.878 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:23.878 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage2 2026-02-24T16:43:23.878 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage2 --load-balancing-group 3 2026-02-24T16:43:23.925 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:23Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-mon[43553]: pgmap v110: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 2.7 KiB/s wr, 78 op/s 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:1684 (105): Start update from 18 to 19 (discovery-trial183). 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:2068 (105): Update complete (18 -> 19) (discovery-trial183). 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 18 to 19 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode3 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (18 -> 19) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:23 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-mon[39453]: pgmap v110: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 2.7 KiB/s wr, 78 op/s 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:1684 (81): Start update from 18 to 19 (discovery-trial151). 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:2068 (81): Update complete (18 -> 19) (discovery-trial151). 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:42494 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage1, no need to remove 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:42494 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.751271] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_1, nonce 10.20.193.151:0/185959448, rc 0 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.751356] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_1, nonce 10.20.193.151:0/185959448 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO grpc.py:1280 (2): Allocated cluster name='cluster_1' nonce='10.20.193.151:0/185959448' 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.760989] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO grpc.py:2535 (2): Received request to add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage1, context: , client address: IPv4 10.20.193.51:42494 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.765736] subsystem_rbd_resrv.c: 52:spdk_try_rbd_reservation_ops_set: *NOTICE*: reservation custom ops set for for bdev_rbd 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.765904] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage1 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.765920] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_1, OMAP version: 20 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.850993] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:23.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:23.851023] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-mon[44017]: pgmap v110: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 2.7 KiB/s wr, 78 op/s 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 18 to 19 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:1684 (132): Start update from 18 to 19 (discovery-trial153). 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:2068 (132): Update complete (18 -> 19) (discovery-trial153). 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode3 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (18 -> 19) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:23 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:23] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:1684 (126): Start update from 18 to 19 (discovery-trial191). 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:2068 (126): Update complete (18 -> 19) (discovery-trial191). 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:1684 (2): Start update from 18 to 19 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO grpc.py:4902 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING grpc.py:4951 (2): Subsystem nqn.2016-06.io.spdk:cnode3 will be opened to be accessed from any host. This might be a security breach 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO state.py:2068 (2): Update complete (18 -> 19) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] INFO grpc.py:2649 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: states { 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: grp_id: 4 2026-02-24T16:43:24.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: state: OPTIMIZED 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: } 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: states { 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: grp_id: 4 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: state: OPTIMIZED 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: } 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: states { 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: grp_id: 4 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: state: OPTIMIZED 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: } 2026-02-24T16:43:24.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: ], , client address: IPv4 10.20.193.191:35878 2026-02-24T16:43:24.093 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.093 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:23 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:23] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.546 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:1684 (81): Start update from 19 to 20 (discovery-trial151). 2026-02-24T16:43:24.546 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:2068 (81): Update complete (19 -> 20) (discovery-trial151). 2026-02-24T16:43:24.546 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:43768 2026-02-24T16:43:24.546 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage2, no need to remove 2026-02-24T16:43:24.546 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:43768 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.451161] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_2, nonce 10.20.193.151:0/2588021982, rc 0 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.451628] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_2, nonce 10.20.193.151:0/2588021982 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO grpc.py:1280 (2): Allocated cluster name='cluster_2' nonce='10.20.193.151:0/2588021982' 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.460555] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO grpc.py:2535 (2): Received request to add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage2, context: , client address: IPv4 10.20.193.51:43768 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.465290] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage2 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.465311] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_2, OMAP version: 21 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.548254] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:24.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:24.548280] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:24.548 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:24.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 19 to 20 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:24.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:24.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:1684 (132): Start update from 19 to 20 (discovery-trial153). 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:2068 (132): Update complete (19 -> 20) (discovery-trial153). 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.320054] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_1, nonce 10.20.193.153:0/3270955743, rc 0 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.320137] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_1, nonce 10.20.193.153:0/3270955743 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:1280 (2): Allocated cluster name='cluster_1' nonce='10.20.193.153:0/3270955743' 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.330757] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:2535 (2): Received request to add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage1, context: None 2026-02-24T16:43:24.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.335423] subsystem_rbd_resrv.c: 52:spdk_try_rbd_reservation_ops_set: *NOTICE*: reservation custom ops set for for bdev_rbd 2026-02-24T16:43:24.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.335626] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage1 2026-02-24T16:43:24.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:24.335641] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:24.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:2068 (2): Update complete (19 -> 20) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:24.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.567 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:24.567 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:24.567 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage3 2026-02-24T16:43:24.567 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage3 --load-balancing-group 4 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:1684 (126): Start update from 19 to 20 (discovery-trial191). 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:2068 (126): Update complete (19 -> 20) (discovery-trial191). 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 19 to 20 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.323441] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_1, nonce 10.20.193.191:0/3039741901, rc 0 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.323518] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_1, nonce 10.20.193.191:0/3039741901 2026-02-24T16:43:24.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:1280 (2): Allocated cluster name='cluster_1' nonce='10.20.193.191:0/3039741901' 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.334165] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:2535 (2): Received request to add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage1, context: None 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.338850] subsystem_rbd_resrv.c: 52:spdk_try_rbd_reservation_ops_set: *NOTICE*: reservation custom ops set for for bdev_rbd 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.339055] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage1 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:24.339070] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:2068 (2): Update complete (19 -> 20) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.602 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:24Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:24.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:1684 (105): Start update from 19 to 20 (discovery-trial183). 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:2068 (105): Update complete (19 -> 20) (discovery-trial183). 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 19 to 20 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.317158] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_1, nonce 10.20.193.183:0/2448160089, rc 0 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.317250] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_1, nonce 10.20.193.183:0/2448160089 2026-02-24T16:43:24.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:1280 (2): Allocated cluster name='cluster_1' nonce='10.20.193.183:0/2448160089' 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.326997] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:2535 (2): Received request to add bdev_840516f4-691c-4ba5-a1ba-efa5fa20e5a1 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage1, context: None 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.331629] subsystem_rbd_resrv.c: 52:spdk_try_rbd_reservation_ops_set: *NOTICE*: reservation custom ops set for for bdev_rbd 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.331835] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage1 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.331851] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:2068 (2): Update complete (19 -> 20) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:24.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.230 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:1684 (81): Start update from 20 to 21 (discovery-trial151). 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:24] INFO state.py:2068 (81): Update complete (20 -> 21) (discovery-trial151). 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:57128 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage3, no need to remove 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40fce57c-8276-4a72-b45a-7be331c8e74f from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:57128 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.117386] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_3, nonce 10.20.193.151:0/1192518306, rc 0 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.121635] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_3, nonce 10.20.193.151:0/1192518306 2026-02-24T16:43:25.231 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_3' nonce='10.20.193.151:0/1192518306' 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.139633] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage3, context: , client address: IPv4 10.20.193.51:57128 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.145832] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage3 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.145855] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_3, OMAP version: 22 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.232886] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:25.232 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.232910] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:25.233 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:25.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:1684 (105): Start update from 20 to 21 (discovery-trial183). 2026-02-24T16:43:25.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:2068 (105): Update complete (20 -> 21) (discovery-trial183). 2026-02-24T16:43:25.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 20 to 21 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:25.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:24 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:24.999582] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_2, nonce 10.20.193.183:0/3074053471, rc 0 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.000244] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_2, nonce 10.20.193.183:0/3074053471 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_2' nonce='10.20.193.183:0/3074053471' 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.009571] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage2, context: None 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.014268] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage2 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.014290] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (20 -> 21) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.254 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:25.254 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:25.254 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage4 2026-02-24T16:43:25.254 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage4 --load-balancing-group 1 2026-02-24T16:43:25.297 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:25Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:25.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:1684 (132): Start update from 20 to 21 (discovery-trial153). 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:2068 (132): Update complete (20 -> 21) (discovery-trial153). 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 20 to 21 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:24 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.002983] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_2, nonce 10.20.193.153:0/3422923923, rc 0 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.003046] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_2, nonce 10.20.193.153:0/3422923923 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_2' nonce='10.20.193.153:0/3422923923' 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.012585] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage2, context: None 2026-02-24T16:43:25.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.017454] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage2 2026-02-24T16:43:25.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.017474] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:25.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (20 -> 21) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:25.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:1684 (126): Start update from 20 to 21 (discovery-trial191). 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:2068 (126): Update complete (20 -> 21) (discovery-trial191). 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO state.py:1684 (2): Start update from 20 to 21 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:24] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.011288] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_2, nonce 10.20.193.191:0/3848634005, rc 0 2026-02-24T16:43:25.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.011351] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_2, nonce 10.20.193.191:0/3848634005 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_2' nonce='10.20.193.191:0/3848634005' 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.019902] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_cfb31ce7-cb91-4b97-adf8-8272506ab870 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage2, context: None 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.024761] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage2 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.024785] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (20 -> 21) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.956 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-mon[39453]: pgmap v111: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 66 KiB/s rd, 2.3 KiB/s wr, 70 op/s 2026-02-24T16:43:25.956 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3608977199' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:1684 (81): Start update from 21 to 22 (discovery-trial151). 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:2068 (81): Update complete (21 -> 22) (discovery-trial151). 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:56394 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage4, no need to remove 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:56394 2026-02-24T16:43:25.957 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.853985] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_4, nonce 10.20.193.151:0/1127846164, rc 0 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.854078] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_4, nonce 10.20.193.151:0/1127846164 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_4' nonce='10.20.193.151:0/1127846164' 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.862718] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage4, context: , client address: IPv4 10.20.193.51:56394 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.867236] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage4 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.867258] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_4, OMAP version: 23 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:25] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.958775] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:25.958 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:25.958803] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:25.959 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:25.979 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:25.979 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:25.980 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage5 2026-02-24T16:43:25.980 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage5 --load-balancing-group 2 2026-02-24T16:43:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-mon[43553]: pgmap v111: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 66 KiB/s rd, 2.3 KiB/s wr, 70 op/s 2026-02-24T16:43:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3608977199' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO state.py:1684 (2): Start update from 21 to 22 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40fce57c-8276-4a72-b45a-7be331c8e74f from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO state.py:1684 (105): Start update from 21 to 22 (discovery-trial183). 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO state.py:2068 (105): Update complete (21 -> 22) (discovery-trial183). 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.681837] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_3, nonce 10.20.193.183:0/2387339634, rc 0 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.681893] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_3, nonce 10.20.193.183:0/2387339634 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_3' nonce='10.20.193.183:0/2387339634' 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.691691] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage3, context: None 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.696224] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage3 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:25.696247] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (21 -> 22) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:25.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:25.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:25 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.017 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:26.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-mon[44017]: pgmap v111: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 66 KiB/s rd, 2.3 KiB/s wr, 70 op/s 2026-02-24T16:43:26.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3608977199' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO state.py:1684 (2): Start update from 21 to 22 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40fce57c-8276-4a72-b45a-7be331c8e74f from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO state.py:1684 (132): Start update from 21 to 22 (discovery-trial153). 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO state.py:2068 (132): Update complete (21 -> 22) (discovery-trial153). 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.677071] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_3, nonce 10.20.193.153:0/3243388961, rc 0 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.677131] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_3, nonce 10.20.193.153:0/3243388961 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_3' nonce='10.20.193.153:0/3243388961' 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.686744] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage3, context: None 2026-02-24T16:43:26.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.691233] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage3 2026-02-24T16:43:26.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:25.691253] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:26.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (21 -> 22) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:26.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:25 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:25] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO state.py:1684 (126): Start update from 21 to 22 (discovery-trial191). 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO state.py:2068 (126): Update complete (21 -> 22) (discovery-trial191). 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO state.py:1684 (2): Start update from 21 to 22 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40fce57c-8276-4a72-b45a-7be331c8e74f from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:26.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.689057] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_3, nonce 10.20.193.191:0/3657576300, rc 0 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.689130] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_3, nonce 10.20.193.191:0/3657576300 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:1280 (2): Allocated cluster name='cluster_3' nonce='10.20.193.191:0/3657576300' 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.699154] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO grpc.py:2535 (2): Received request to add bdev_40fce57c-8276-4a72-b45a-7be331c8e74f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage3, context: None 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.703682] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage3 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:25.703704] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] INFO state.py:2068 (2): Update complete (21 -> 22) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:25 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:25] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO state.py:1684 (81): Start update from 22 to 23 (discovery-trial151). 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO state.py:2068 (81): Update complete (22 -> 23) (discovery-trial151). 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:59018 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage5, no need to remove 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:26.652 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:59018 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.563008] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_5, nonce 10.20.193.151:0/3461547953, rc 0 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.563635] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_5, nonce 10.20.193.151:0/3461547953 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO grpc.py:1280 (2): Allocated cluster name='cluster_5' nonce='10.20.193.151:0/3461547953' 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.578634] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO grpc.py:2535 (2): Received request to add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage5, context: , client address: IPv4 10.20.193.51:59018 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.604793] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage5 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.604824] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_5, OMAP version: 24 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.653 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.654 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO state.py:1684 (2): Start update from 22 to 23 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:26.654 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:26.654 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:26.654 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO state.py:1684 (132): Start update from 22 to 23 (discovery-trial153). 2026-02-24T16:43:26.654 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO state.py:2068 (132): Update complete (22 -> 23) (discovery-trial153). 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:26.402897] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_4, nonce 10.20.193.153:0/2820410667, rc 0 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:26.402964] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_4, nonce 10.20.193.153:0/2820410667 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO grpc.py:1280 (2): Allocated cluster name='cluster_4' nonce='10.20.193.153:0/2820410667' 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:26.410314] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO grpc.py:2535 (2): Received request to add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage4, context: None 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:26.415194] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage4 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:26.415219] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] INFO state.py:2068 (2): Update complete (22 -> 23) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO state.py:1684 (105): Start update from 22 to 23 (discovery-trial183). 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO state.py:2068 (105): Update complete (22 -> 23) (discovery-trial183). 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO state.py:1684 (2): Start update from 22 to 23 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:26.403778] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_4, nonce 10.20.193.183:0/465417080, rc 0 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:26.403837] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_4, nonce 10.20.193.183:0/465417080 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO grpc.py:1280 (2): Allocated cluster name='cluster_4' nonce='10.20.193.183:0/465417080' 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:26.412802] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO grpc.py:2535 (2): Received request to add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage4, context: None 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:26.417301] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage4 2026-02-24T16:43:26.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:26.417322] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:26.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] INFO state.py:2068 (2): Update complete (22 -> 23) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:26.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.699 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:26.724 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:26.724 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:26.724 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage6 2026-02-24T16:43:26.725 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage6 --load-balancing-group 3 2026-02-24T16:43:26.769 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:26.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO state.py:1684 (126): Start update from 22 to 23 (discovery-trial191). 2026-02-24T16:43:26.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO state.py:2068 (126): Update complete (22 -> 23) (discovery-trial191). 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO state.py:1684 (2): Start update from 22 to 23 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:26.411832] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_4, nonce 10.20.193.191:0/1984338728, rc 0 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:26.411889] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_4, nonce 10.20.193.191:0/1984338728 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO grpc.py:1280 (2): Allocated cluster name='cluster_4' nonce='10.20.193.191:0/1984338728' 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:26.431310] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO grpc.py:2535 (2): Received request to add bdev_0d16b116-783f-43cc-a1cb-37afe0fd7c76 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage4, context: None 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:26.449503] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage4 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:26.449533] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] INFO state.py:2068 (2): Update complete (22 -> 23) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:26.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3899408008' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:26.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3899408008' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:26.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.699553] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:26.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:26.699592] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:27.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3899408008' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:27.382 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:1684 (81): Start update from 23 to 24 (discovery-trial151). 2026-02-24T16:43:27.382 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:2068 (81): Update complete (23 -> 24) (discovery-trial151). 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:33834 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage6, no need to remove 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:33834 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.292531] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_6, nonce 10.20.193.151:0/1877051361, rc 0 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.292596] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_6, nonce 10.20.193.151:0/1877051361 2026-02-24T16:43:27.383 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_6' nonce='10.20.193.151:0/1877051361' 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.301064] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage6, context: , client address: IPv4 10.20.193.51:33834 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.305984] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage6 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.306006] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_6, OMAP version: 25 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.384731] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:27.384 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.384756] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:27.385 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:27.405 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:27.405 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:27.405 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage7 2026-02-24T16:43:27.405 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage7 --load-balancing-group 4 2026-02-24T16:43:27.451 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:27.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:1684 (105): Start update from 23 to 24 (discovery-trial183). 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:2068 (105): Update complete (23 -> 24) (discovery-trial183). 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 23 to 24 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.144967] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_5, nonce 10.20.193.183:0/2040976780, rc 0 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.145029] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_5, nonce 10.20.193.183:0/2040976780 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_5' nonce='10.20.193.183:0/2040976780' 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.154482] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage5, context: None 2026-02-24T16:43:27.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.158908] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage5 2026-02-24T16:43:27.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.158931] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:27.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (23 -> 24) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:27.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:1684 (132): Start update from 23 to 24 (discovery-trial153). 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:2068 (132): Update complete (23 -> 24) (discovery-trial153). 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 23 to 24 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.142755] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_5, nonce 10.20.193.153:0/1778858875, rc 0 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.142812] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_5, nonce 10.20.193.153:0/1778858875 2026-02-24T16:43:27.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_5' nonce='10.20.193.153:0/1778858875' 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.152221] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage5, context: None 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.156767] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage5 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.156787] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (23 -> 24) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:1684 (126): Start update from 23 to 24 (discovery-trial191). 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:2068 (126): Update complete (23 -> 24) (discovery-trial191). 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 23 to 24 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.149782] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_5, nonce 10.20.193.191:0/1609503540, rc 0 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.149836] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_5, nonce 10.20.193.191:0/1609503540 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_5' nonce='10.20.193.191:0/1609503540' 2026-02-24T16:43:27.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.159697] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_b3b53051-701d-48ff-aa3e-45a9ad76db11 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage5, context: None 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.164208] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage5 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.164230] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (23 -> 24) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.941 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-mon[39453]: pgmap v112: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 190 KiB/s rd, 2.8 KiB/s wr, 223 op/s 2026-02-24T16:43:27.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:1684 (81): Start update from 24 to 25 (discovery-trial151). 2026-02-24T16:43:27.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:2068 (81): Update complete (24 -> 25) (discovery-trial151). 2026-02-24T16:43:27.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:38274 2026-02-24T16:43:27.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage7, no need to remove 2026-02-24T16:43:27.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:27.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:27.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fbfcd850-798d-4a37-bc95-da6914211a74 from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:38274 2026-02-24T16:43:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-mon[43553]: pgmap v112: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 190 KiB/s rd, 2.8 KiB/s wr, 223 op/s 2026-02-24T16:43:27.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 24 to 25 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:1684 (105): Start update from 24 to 25 (discovery-trial183). 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:2068 (105): Update complete (24 -> 25) (discovery-trial183). 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.842356] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_6, nonce 10.20.193.183:0/2975139448, rc 0 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.842411] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_6, nonce 10.20.193.183:0/2975139448 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_6' nonce='10.20.193.183:0/2975139448' 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.851389] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage6, context: None 2026-02-24T16:43:27.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.855988] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage6 2026-02-24T16:43:27.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:27.856010] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:27.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (24 -> 25) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:27.988 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-mon[44017]: pgmap v112: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 190 KiB/s rd, 2.8 KiB/s wr, 223 op/s 2026-02-24T16:43:27.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 24 to 25 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:27.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:1684 (132): Start update from 24 to 25 (discovery-trial153). 2026-02-24T16:43:27.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:2068 (132): Update complete (24 -> 25) (discovery-trial153). 2026-02-24T16:43:27.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.844170] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_6, nonce 10.20.193.153:0/3796120272, rc 0 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.844230] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_6, nonce 10.20.193.153:0/3796120272 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_6' nonce='10.20.193.153:0/3796120272' 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.852817] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage6, context: None 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.857781] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage6 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:27.857803] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (24 -> 25) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:27.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.080 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:28.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:1684 (126): Start update from 24 to 25 (discovery-trial191). 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:2068 (126): Update complete (24 -> 25) (discovery-trial191). 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:1684 (2): Start update from 24 to 25 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.852182] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_6, nonce 10.20.193.191:0/3040375331, rc 0 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.852234] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_6, nonce 10.20.193.191:0/3040375331 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_6' nonce='10.20.193.191:0/3040375331' 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.860368] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_e988432a-1296-42f2-b6da-fcaadbb7d5f9 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage6, context: None 2026-02-24T16:43:28.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.865296] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage6 2026-02-24T16:43:28.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:27.865318] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:28.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] INFO state.py:2068 (2): Update complete (24 -> 25) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:28.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.103 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:28.103 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:28.103 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage8 2026-02-24T16:43:28.103 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage8 --load-balancing-group 1 2026-02-24T16:43:28.140 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:28.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.969616] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_7, nonce 10.20.193.151:0/830731432, rc 0 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.969683] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_7, nonce 10.20.193.151:0/830731432 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:1280 (2): Allocated cluster name='cluster_7' nonce='10.20.193.151:0/830731432' 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.979223] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO grpc.py:2535 (2): Received request to add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage7, context: , client address: IPv4 10.20.193.51:38274 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.983766] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage7 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:27.983788] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_7, OMAP version: 26 2026-02-24T16:43:28.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:27] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.080601] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:28.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.080629] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:28.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.778 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/814693701' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.778 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4041506176' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO state.py:1684 (81): Start update from 25 to 26 (discovery-trial151). 2026-02-24T16:43:28.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO state.py:2068 (81): Update complete (25 -> 26) (discovery-trial151). 2026-02-24T16:43:28.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:58550 2026-02-24T16:43:28.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage8, no need to remove 2026-02-24T16:43:28.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:58550 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.664681] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_8, nonce 10.20.193.151:0/3529451373, rc 0 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.665627] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_8, nonce 10.20.193.151:0/3529451373 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO grpc.py:1280 (2): Allocated cluster name='cluster_8' nonce='10.20.193.151:0/3529451373' 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.682633] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO grpc.py:2535 (2): Received request to add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage8, context: , client address: IPv4 10.20.193.51:58550 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.699781] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage8 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.699812] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_8, OMAP version: 27 2026-02-24T16:43:28.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.781 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.781 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.781461] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:28.781 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:28.781493] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:28.781 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:28.802 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:28.802 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:28.802 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage9 2026-02-24T16:43:28.802 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage9 --load-balancing-group 2 2026-02-24T16:43:28.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/814693701' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4041506176' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO state.py:1684 (132): Start update from 25 to 26 (discovery-trial153). 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO state.py:2068 (132): Update complete (25 -> 26) (discovery-trial153). 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO state.py:1684 (2): Start update from 25 to 26 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fbfcd850-798d-4a37-bc95-da6914211a74 from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:28.521440] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_7, nonce 10.20.193.153:0/3599829468, rc 0 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:28.521511] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_7, nonce 10.20.193.153:0/3599829468 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO grpc.py:1280 (2): Allocated cluster name='cluster_7' nonce='10.20.193.153:0/3599829468' 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:28.531253] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO grpc.py:2535 (2): Received request to add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage7, context: None 2026-02-24T16:43:28.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:28.535976] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage7 2026-02-24T16:43:28.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:28.535998] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:28.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] INFO state.py:2068 (2): Update complete (25 -> 26) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:28.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:28] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO state.py:1684 (126): Start update from 25 to 26 (discovery-trial191). 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO state.py:2068 (126): Update complete (25 -> 26) (discovery-trial191). 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO state.py:1684 (2): Start update from 25 to 26 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fbfcd850-798d-4a37-bc95-da6914211a74 from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:28.533073] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_7, nonce 10.20.193.191:0/2105066843, rc 0 2026-02-24T16:43:28.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:28.533130] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_7, nonce 10.20.193.191:0/2105066843 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO grpc.py:1280 (2): Allocated cluster name='cluster_7' nonce='10.20.193.191:0/2105066843' 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:28.542395] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO grpc.py:2535 (2): Received request to add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage7, context: None 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:28.547199] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage7 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:28.547222] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] INFO state.py:2068 (2): Update complete (25 -> 26) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.848 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:28.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/814693701' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4041506176' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO state.py:1684 (105): Start update from 25 to 26 (discovery-trial183). 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO state.py:2068 (105): Update complete (25 -> 26) (discovery-trial183). 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO state.py:1684 (2): Start update from 25 to 26 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fbfcd850-798d-4a37-bc95-da6914211a74 from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:28.523624] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_7, nonce 10.20.193.183:0/2873618352, rc 0 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:28.524266] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_7, nonce 10.20.193.183:0/2873618352 2026-02-24T16:43:28.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO grpc.py:1280 (2): Allocated cluster name='cluster_7' nonce='10.20.193.183:0/2873618352' 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:28.533470] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO grpc.py:2535 (2): Received request to add bdev_fbfcd850-798d-4a37-bc95-da6914211a74 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage7, context: None 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:28.538065] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage7 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:28.538086] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] INFO state.py:2068 (2): Update complete (25 -> 26) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:28.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:1684 (105): Start update from 26 to 27 (discovery-trial183). 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:2068 (105): Update complete (26 -> 27) (discovery-trial183). 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 26 to 27 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.237612] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_8, nonce 10.20.193.183:0/1638179712, rc 0 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.237676] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_8, nonce 10.20.193.183:0/1638179712 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_8' nonce='10.20.193.183:0/1638179712' 2026-02-24T16:43:29.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.246907] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage8, context: None 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.251629] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage8 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.251652] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:2068 (2): Update complete (26 -> 27) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:1684 (81): Start update from 26 to 27 (discovery-trial151). 2026-02-24T16:43:29.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:2068 (81): Update complete (26 -> 27) (discovery-trial151). 2026-02-24T16:43:29.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:56942 2026-02-24T16:43:29.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage9, no need to remove 2026-02-24T16:43:29.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:56942 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.393360] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_9, nonce 10.20.193.151:0/2820278102, rc 0 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.393419] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_9, nonce 10.20.193.151:0/2820278102 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_9' nonce='10.20.193.151:0/2820278102' 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.403339] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage9, context: , client address: IPv4 10.20.193.51:56942 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.407988] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage9 2026-02-24T16:43:29.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.408009] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:29.488 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_9, OMAP version: 28 2026-02-24T16:43:29.488 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.488 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.505 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:29.527 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:29.527 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:29.527 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage10 2026-02-24T16:43:29.527 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage10 --load-balancing-group 3 2026-02-24T16:43:29.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 26 to 27 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:1684 (132): Start update from 26 to 27 (discovery-trial153). 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:2068 (132): Update complete (26 -> 27) (discovery-trial153). 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.231839] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_8, nonce 10.20.193.153:0/2363390265, rc 0 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.232432] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_8, nonce 10.20.193.153:0/2363390265 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_8' nonce='10.20.193.153:0/2363390265' 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.240916] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:29.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage8, context: None 2026-02-24T16:43:29.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.245419] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage8 2026-02-24T16:43:29.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.245440] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:29.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:2068 (2): Update complete (26 -> 27) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:29.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.567 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:29Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:29.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:1684 (126): Start update from 26 to 27 (discovery-trial191). 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:2068 (126): Update complete (26 -> 27) (discovery-trial191). 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 26 to 27 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.247192] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_8, nonce 10.20.193.191:0/4123018386, rc 0 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.247249] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_8, nonce 10.20.193.191:0/4123018386 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_8' nonce='10.20.193.191:0/4123018386' 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.256167] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:29.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_1487f5af-dbee-446c-8277-9e3e8cc61d9a to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage8, context: None 2026-02-24T16:43:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.260676] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage8 2026-02-24T16:43:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.260697] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:2068 (2): Update complete (26 -> 27) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:29.916 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-mon[44017]: pgmap v113: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 300 KiB/s rd, 4.2 KiB/s wr, 333 op/s 2026-02-24T16:43:29.918 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-mon[43553]: pgmap v113: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 300 KiB/s rd, 4.2 KiB/s wr, 333 op/s 2026-02-24T16:43:29.918 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-mon[39453]: pgmap v113: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 300 KiB/s rd, 4.2 KiB/s wr, 333 op/s 2026-02-24T16:43:29.919 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.504829] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:29.919 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:29.504859] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:29.919 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:1684 (81): Start update from 27 to 28 (discovery-trial151). 2026-02-24T16:43:29.919 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:29] INFO state.py:2068 (81): Update complete (27 -> 28) (discovery-trial151). 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 27 to 28 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:1684 (105): Start update from 27 to 28 (discovery-trial183). 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO state.py:2068 (105): Update complete (27 -> 28) (discovery-trial183). 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.945695] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_9, nonce 10.20.193.183:0/4077625635, rc 0 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.946237] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_9, nonce 10.20.193.183:0/4077625635 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_9' nonce='10.20.193.183:0/4077625635' 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.955985] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage9, context: None 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.960444] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage9 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:29.960465] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:30.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (27 -> 28) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:33038 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage10, no need to remove 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:33038 2026-02-24T16:43:30.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.093067] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_10, nonce 10.20.193.151:0/2669878671, rc 0 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.093629] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_10, nonce 10.20.193.151:0/2669878671 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:1280 (2): Allocated cluster name='cluster_10' nonce='10.20.193.151:0/2669878671' 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.102635] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:2535 (2): Received request to add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage10, context: , client address: IPv4 10.20.193.51:33038 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.126806] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage10 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.126828] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_10, OMAP version: 29 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.241 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:30.262 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:30.262 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:30.262 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage11 2026-02-24T16:43:30.262 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage11 --load-balancing-group 4 2026-02-24T16:43:30.300 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:30Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:30.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 27 to 28 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:1684 (132): Start update from 27 to 28 (discovery-trial153). 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO state.py:2068 (132): Update complete (27 -> 28) (discovery-trial153). 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.945346] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_9, nonce 10.20.193.153:0/4010244301, rc 0 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.952433] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_9, nonce 10.20.193.153:0/4010244301 2026-02-24T16:43:30.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_9' nonce='10.20.193.153:0/4010244301' 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.974425] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage9, context: None 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.979697] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage9 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:29.979720] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (27 -> 28) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:1684 (126): Start update from 27 to 28 (discovery-trial191). 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:2068 (126): Update complete (27 -> 28) (discovery-trial191). 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO state.py:1684 (2): Start update from 27 to 28 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.952329] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_9, nonce 10.20.193.191:0/2915077579, rc 0 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.952380] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_9, nonce 10.20.193.191:0/2915077579 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:1280 (2): Allocated cluster name='cluster_9' nonce='10.20.193.191:0/2915077579' 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.961520] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:29] INFO grpc.py:2535 (2): Received request to add bdev_ccdaa0e7-2dd6-4d1c-bc1d-3a9533c7f552 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage9, context: None 2026-02-24T16:43:30.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.966368] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage9 2026-02-24T16:43:30.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:29.966390] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:30.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (27 -> 28) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:30.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.662 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.241614] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:30.663 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.241642] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:30.951 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:1684 (81): Start update from 28 to 29 (discovery-trial151). 2026-02-24T16:43:30.951 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:2068 (81): Update complete (28 -> 29) (discovery-trial151). 2026-02-24T16:43:30.951 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:32912 2026-02-24T16:43:30.951 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage11, no need to remove 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:32912 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.839045] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_11, nonce 10.20.193.151:0/1652215932, rc 0 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.839123] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_11, nonce 10.20.193.151:0/1652215932 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:1280 (2): Allocated cluster name='cluster_11' nonce='10.20.193.151:0/1652215932' 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.849632] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO grpc.py:2535 (2): Received request to add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage11, context: , client address: IPv4 10.20.193.51:32912 2026-02-24T16:43:30.952 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.860806] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage11 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.860826] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_11, OMAP version: 30 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.953415] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:30.953 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:30.953441] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:30.954 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:30.974 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:30.974 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:30.974 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage12 2026-02-24T16:43:30.974 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage12 --load-balancing-group 1 2026-02-24T16:43:30.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO state.py:1684 (2): Start update from 28 to 29 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:30.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:30.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:30.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO state.py:1684 (105): Start update from 28 to 29 (discovery-trial183). 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO state.py:2068 (105): Update complete (28 -> 29) (discovery-trial183). 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:30.687913] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_10, nonce 10.20.193.183:0/3622409234, rc 0 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:30.687969] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_10, nonce 10.20.193.183:0/3622409234 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO grpc.py:1280 (2): Allocated cluster name='cluster_10' nonce='10.20.193.183:0/3622409234' 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:30.697348] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO grpc.py:2535 (2): Received request to add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage10, context: None 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:30.701834] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage10 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:30.701854] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (28 -> 29) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:30] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.014 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:31.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO state.py:1684 (132): Start update from 28 to 29 (discovery-trial153). 2026-02-24T16:43:31.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO state.py:2068 (132): Update complete (28 -> 29) (discovery-trial153). 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO state.py:1684 (2): Start update from 28 to 29 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:30.690389] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_10, nonce 10.20.193.153:0/1264953212, rc 0 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:30.690453] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_10, nonce 10.20.193.153:0/1264953212 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO grpc.py:1280 (2): Allocated cluster name='cluster_10' nonce='10.20.193.153:0/1264953212' 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:30.699196] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO grpc.py:2535 (2): Received request to add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage10, context: None 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:30.703818] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage10 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:30.703842] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (28 -> 29) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO state.py:1684 (126): Start update from 28 to 29 (discovery-trial191). 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO state.py:2068 (126): Update complete (28 -> 29) (discovery-trial191). 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO state.py:1684 (2): Start update from 28 to 29 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:30.698374] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_10, nonce 10.20.193.191:0/523623880, rc 0 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:30.699309] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_10, nonce 10.20.193.191:0/523623880 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO grpc.py:1280 (2): Allocated cluster name='cluster_10' nonce='10.20.193.191:0/523623880' 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:30.707956] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO grpc.py:2535 (2): Received request to add bdev_967ce80d-eb27-4704-9970-b9c49e6ba94e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage10, context: None 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:30.712447] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage10 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:30.712468] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] INFO state.py:2068 (2): Update complete (28 -> 29) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO state.py:1684 (81): Start update from 29 to 30 (discovery-trial151). 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO state.py:2068 (81): Update complete (29 -> 30) (discovery-trial151). 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:44410 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage12, no need to remove 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fcc74cc6-f263-426a-acff-c8c762707e26 from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:44410 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.550035] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_12, nonce 10.20.193.151:0/1974472908, rc 0 2026-02-24T16:43:31.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.550177] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_12, nonce 10.20.193.151:0/1974472908 2026-02-24T16:43:31.649 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:31.665 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO state.py:1684 (2): Start update from 29 to 30 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO state.py:1684 (105): Start update from 29 to 30 (discovery-trial183). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO state.py:2068 (105): Update complete (29 -> 30) (discovery-trial183). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:31.393334] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_11, nonce 10.20.193.183:0/3591296062, rc 0 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:31.393396] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_11, nonce 10.20.193.183:0/3591296062 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO grpc.py:1280 (2): Allocated cluster name='cluster_11' nonce='10.20.193.183:0/3591296062' 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:31.404251] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO grpc.py:2535 (2): Received request to add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage11, context: None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:31.408875] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage11 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:31.408896] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] INFO state.py:2068 (2): Update complete (29 -> 30) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO state.py:1684 (132): Start update from 29 to 30 (discovery-trial153). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO state.py:2068 (132): Update complete (29 -> 30) (discovery-trial153). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO state.py:1684 (2): Start update from 29 to 30 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:31.411215] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_11, nonce 10.20.193.153:0/1831873364, rc 0 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:31.411278] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_11, nonce 10.20.193.153:0/1831873364 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO grpc.py:1280 (2): Allocated cluster name='cluster_11' nonce='10.20.193.153:0/1831873364' 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:31.421415] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO grpc.py:2535 (2): Received request to add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage11, context: None 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:31.426986] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage11 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:31.427009] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] INFO state.py:2068 (2): Update complete (29 -> 30) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.666 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.672 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:31.672 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:31.672 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage13 2026-02-24T16:43:31.672 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage13 --load-balancing-group 2 2026-02-24T16:43:31.711 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:31.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO state.py:1684 (126): Start update from 29 to 30 (discovery-trial191). 2026-02-24T16:43:31.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO state.py:2068 (126): Update complete (29 -> 30) (discovery-trial191). 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO state.py:1684 (2): Start update from 29 to 30 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:31.412469] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_11, nonce 10.20.193.191:0/2158428243, rc 0 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:31.412522] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_11, nonce 10.20.193.191:0/2158428243 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO grpc.py:1280 (2): Allocated cluster name='cluster_11' nonce='10.20.193.191:0/2158428243' 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:31.422107] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO grpc.py:2535 (2): Received request to add bdev_b1ff6657-0bd2-4a14-a224-d9518b9e0e8b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage11, context: None 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:31.426593] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage11 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:31.426616] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] INFO state.py:2068 (2): Update complete (29 -> 30) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:31] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:31 trial183 ceph-mon[43553]: pgmap v114: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 298 KiB/s rd, 2.7 KiB/s wr, 334 op/s 2026-02-24T16:43:31.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-mon[39453]: pgmap v114: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 298 KiB/s rd, 2.7 KiB/s wr, 334 op/s 2026-02-24T16:43:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO grpc.py:1280 (2): Allocated cluster name='cluster_12' nonce='10.20.193.151:0/1974472908' 2026-02-24T16:43:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.558699] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO grpc.py:2535 (2): Received request to add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage12, context: , client address: IPv4 10.20.193.51:44410 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.563264] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage12 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.563287] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_12, OMAP version: 31 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.649150] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:31.649183] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:32.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:31 trial153 ceph-mon[44017]: pgmap v114: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 298 KiB/s rd, 2.7 KiB/s wr, 334 op/s 2026-02-24T16:43:32.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 30 to 31 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:32.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fcc74cc6-f263-426a-acff-c8c762707e26 from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:1684 (126): Start update from 30 to 31 (discovery-trial191). 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:2068 (126): Update complete (30 -> 31) (discovery-trial191). 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.106495] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_12, nonce 10.20.193.191:0/3329591542, rc 0 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.106550] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_12, nonce 10.20.193.191:0/3329591542 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_12' nonce='10.20.193.191:0/3329591542' 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.116041] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage12, context: None 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.120835] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage12 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.120856] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (30 -> 31) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:1684 (81): Start update from 30 to 31 (discovery-trial151). 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:2068 (81): Update complete (30 -> 31) (discovery-trial151). 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:64653 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage13, no need to remove 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:64653 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.255170] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_13, nonce 10.20.193.151:0/2315310792, rc 0 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.255236] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_13, nonce 10.20.193.151:0/2315310792 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_13' nonce='10.20.193.151:0/2315310792' 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.264858] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage13, context: , client address: IPv4 10.20.193.51:64653 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.269374] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage13 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.269395] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_13, OMAP version: 32 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.354039] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:32.351 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.354065] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:32.353 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:32.375 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:32.375 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:32.375 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage14 2026-02-24T16:43:32.375 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage14 --load-balancing-group 3 2026-02-24T16:43:32.415 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:32Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:32.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 30 to 31 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fcc74cc6-f263-426a-acff-c8c762707e26 from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:1684 (105): Start update from 30 to 31 (discovery-trial183). 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:2068 (105): Update complete (30 -> 31) (discovery-trial183). 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.100324] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_12, nonce 10.20.193.183:0/294136694, rc 0 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.100383] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_12, nonce 10.20.193.183:0/294136694 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_12' nonce='10.20.193.183:0/294136694' 2026-02-24T16:43:32.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.110101] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage12, context: None 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.114917] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage12 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.114937] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (30 -> 31) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:1684 (132): Start update from 30 to 31 (discovery-trial153). 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:2068 (132): Update complete (30 -> 31) (discovery-trial153). 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 30 to 31 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_fcc74cc6-f263-426a-acff-c8c762707e26 from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.102207] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_12, nonce 10.20.193.153:0/1957668147, rc 0 2026-02-24T16:43:32.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.102272] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_12, nonce 10.20.193.153:0/1957668147 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_12' nonce='10.20.193.153:0/1957668147' 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.111208] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_fcc74cc6-f263-426a-acff-c8c762707e26 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage12, context: None 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.115907] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage12 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.115928] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (30 -> 31) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.925 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/201506683' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:1684 (81): Start update from 31 to 32 (discovery-trial151). 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:2068 (81): Update complete (31 -> 32) (discovery-trial151). 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:60410 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage14, no need to remove 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:32.926 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:60410 2026-02-24T16:43:32.968 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/201506683' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:32.968 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:1684 (105): Start update from 31 to 32 (discovery-trial183). 2026-02-24T16:43:32.968 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:2068 (105): Update complete (31 -> 32) (discovery-trial183). 2026-02-24T16:43:32.968 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 31 to 32 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.806497] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_13, nonce 10.20.193.183:0/1876298088, rc 0 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.806549] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_13, nonce 10.20.193.183:0/1876298088 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_13' nonce='10.20.193.183:0/1876298088' 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.815610] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage13, context: None 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.820221] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage13 2026-02-24T16:43:32.969 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:32.820240] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:32.970 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (31 -> 32) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:32.970 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:32.970 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/201506683' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:1684 (132): Start update from 31 to 32 (discovery-trial153). 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:2068 (132): Update complete (31 -> 32) (discovery-trial153). 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 31 to 32 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.811502] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_13, nonce 10.20.193.153:0/3321372169, rc 0 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.811565] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_13, nonce 10.20.193.153:0/3321372169 2026-02-24T16:43:32.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_13' nonce='10.20.193.153:0/3321372169' 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.820422] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage13, context: None 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.824960] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage13 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:32.824982] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (31 -> 32) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:32.972 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.050 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:33.071 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:33.071 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:33.071 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage15 2026-02-24T16:43:33.071 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage15 --load-balancing-group 4 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:1684 (126): Start update from 31 to 32 (discovery-trial191). 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:2068 (126): Update complete (31 -> 32) (discovery-trial191). 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:1684 (2): Start update from 31 to 32 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.816893] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_13, nonce 10.20.193.191:0/566954486, rc 0 2026-02-24T16:43:33.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.816958] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_13, nonce 10.20.193.191:0/566954486 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_13' nonce='10.20.193.191:0/566954486' 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.826012] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_2d6ea4fa-a156-4fb6-b260-22c7c4ea52d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage13, context: None 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.830454] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage13 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:32.830475] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] INFO state.py:2068 (2): Update complete (31 -> 32) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.110 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:33Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:33.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.950074] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_14, nonce 10.20.193.151:0/1779359085, rc 0 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.950139] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_14, nonce 10.20.193.151:0/1779359085 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:1280 (2): Allocated cluster name='cluster_14' nonce='10.20.193.151:0/1779359085' 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.959871] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO grpc.py:2535 (2): Received request to add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage14, context: , client address: IPv4 10.20.193.51:60410 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.964431] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage14 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:32.964452] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_14, OMAP version: 33 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:32] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.050690] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.050714] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:33.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-mon[43553]: pgmap v115: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 4.9 KiB/s wr, 749 op/s 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO state.py:1684 (105): Start update from 32 to 33 (discovery-trial183). 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO state.py:2068 (105): Update complete (32 -> 33) (discovery-trial183). 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO state.py:1684 (2): Start update from 32 to 33 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:33.503876] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_14, nonce 10.20.193.183:0/2335855432, rc 0 2026-02-24T16:43:33.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:33.503933] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_14, nonce 10.20.193.183:0/2335855432 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO grpc.py:1280 (2): Allocated cluster name='cluster_14' nonce='10.20.193.183:0/2335855432' 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:33.514265] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO grpc.py:2535 (2): Received request to add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage14, context: None 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:33.518850] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage14 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:33.518872] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] INFO state.py:2068 (2): Update complete (32 -> 33) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.736 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-mon[39453]: pgmap v115: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 4.9 KiB/s wr, 749 op/s 2026-02-24T16:43:33.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO state.py:1684 (81): Start update from 32 to 33 (discovery-trial151). 2026-02-24T16:43:33.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO state.py:2068 (81): Update complete (32 -> 33) (discovery-trial151). 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:35448 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage15, no need to remove 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_047869d6-7b93-44d0-8cef-79878331cc42 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:35448 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.648267] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_15, nonce 10.20.193.151:0/4293199874, rc 0 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.648351] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_15, nonce 10.20.193.151:0/4293199874 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO grpc.py:1280 (2): Allocated cluster name='cluster_15' nonce='10.20.193.151:0/4293199874' 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.657395] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_047869d6-7b93-44d0-8cef-79878331cc42 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.737 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO grpc.py:2535 (2): Received request to add bdev_047869d6-7b93-44d0-8cef-79878331cc42 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage15, context: , client address: IPv4 10.20.193.51:35448 2026-02-24T16:43:33.738 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.661890] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage15 2026-02-24T16:43:33.738 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.661912] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:33.738 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_15, OMAP version: 34 2026-02-24T16:43:33.738 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.738 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:33] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.744 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:33.771 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:33.771 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:33.771 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage16 2026-02-24T16:43:33.771 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage16 --load-balancing-group 1 2026-02-24T16:43:33.809 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:33Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:33.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-mon[44017]: pgmap v115: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 4.9 KiB/s wr, 749 op/s 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO state.py:1684 (132): Start update from 32 to 33 (discovery-trial153). 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO state.py:2068 (132): Update complete (32 -> 33) (discovery-trial153). 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO state.py:1684 (2): Start update from 32 to 33 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:33.500934] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_14, nonce 10.20.193.153:0/1406574996, rc 0 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:33.501003] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_14, nonce 10.20.193.153:0/1406574996 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO grpc.py:1280 (2): Allocated cluster name='cluster_14' nonce='10.20.193.153:0/1406574996' 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:33.509890] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO grpc.py:2535 (2): Received request to add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage14, context: None 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:33.514406] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage14 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:33.514427] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] INFO state.py:2068 (2): Update complete (32 -> 33) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO state.py:1684 (126): Start update from 32 to 33 (discovery-trial191). 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO state.py:2068 (126): Update complete (32 -> 33) (discovery-trial191). 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO state.py:1684 (2): Start update from 32 to 33 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:33.510367] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_14, nonce 10.20.193.191:0/607186255, rc 0 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:33.510431] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_14, nonce 10.20.193.191:0/607186255 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO grpc.py:1280 (2): Allocated cluster name='cluster_14' nonce='10.20.193.191:0/607186255' 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:33.520020] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO grpc.py:2535 (2): Received request to add bdev_a8513d36-6a5a-4fb4-86c4-a114c780b45e to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage14, context: None 2026-02-24T16:43:33.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:33.524543] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage14 2026-02-24T16:43:33.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:33.524564] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:33.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] INFO state.py:2068 (2): Update complete (32 -> 33) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:33.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:33.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.744718] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:34.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:33.744749] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:34.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:1684 (81): Start update from 33 to 34 (discovery-trial151). 2026-02-24T16:43:34.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:2068 (81): Update complete (33 -> 34) (discovery-trial151). 2026-02-24T16:43:34.431 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:55388 2026-02-24T16:43:34.431 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage16, no need to remove 2026-02-24T16:43:34.431 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:34.431 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:34.431 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:55388 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.333637] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_16, nonce 10.20.193.151:0/3344556631, rc 0 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.333705] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_16, nonce 10.20.193.151:0/3344556631 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_16' nonce='10.20.193.151:0/3344556631' 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.342785] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage16, context: , client address: IPv4 10.20.193.51:55388 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.347405] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage16 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.347428] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_16, OMAP version: 35 2026-02-24T16:43:34.432 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.433 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.433 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.433566] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:34.433 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:34.433592] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:34.433 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:34.454 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:34.454 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:34.454 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage17 2026-02-24T16:43:34.454 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage17 --load-balancing-group 2 2026-02-24T16:43:34.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:1684 (105): Start update from 33 to 34 (discovery-trial183). 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:2068 (105): Update complete (33 -> 34) (discovery-trial183). 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 33 to 34 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_047869d6-7b93-44d0-8cef-79878331cc42 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.198507] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_15, nonce 10.20.193.183:0/2195786928, rc 0 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.198564] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_15, nonce 10.20.193.183:0/2195786928 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_15' nonce='10.20.193.183:0/2195786928' 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.207904] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_047869d6-7b93-44d0-8cef-79878331cc42 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_047869d6-7b93-44d0-8cef-79878331cc42 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage15, context: None 2026-02-24T16:43:34.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.212541] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage15 2026-02-24T16:43:34.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.212563] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:34.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (33 -> 34) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:34.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.494 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:34Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:34.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:1684 (132): Start update from 33 to 34 (discovery-trial153). 2026-02-24T16:43:34.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:2068 (132): Update complete (33 -> 34) (discovery-trial153). 2026-02-24T16:43:34.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 33 to 34 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:34.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_047869d6-7b93-44d0-8cef-79878331cc42 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.202190] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_15, nonce 10.20.193.153:0/2642749458, rc 0 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.202252] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_15, nonce 10.20.193.153:0/2642749458 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_15' nonce='10.20.193.153:0/2642749458' 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.211665] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_047869d6-7b93-44d0-8cef-79878331cc42 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_047869d6-7b93-44d0-8cef-79878331cc42 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage15, context: None 2026-02-24T16:43:34.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.216238] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage15 2026-02-24T16:43:34.566 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.216260] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:34.566 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (33 -> 34) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:34.566 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.566 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:1684 (126): Start update from 33 to 34 (discovery-trial191). 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:2068 (126): Update complete (33 -> 34) (discovery-trial191). 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 33 to 34 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_047869d6-7b93-44d0-8cef-79878331cc42 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.208540] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_15, nonce 10.20.193.191:0/3623927670, rc 0 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.208601] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_15, nonce 10.20.193.191:0/3623927670 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_15' nonce='10.20.193.191:0/3623927670' 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.218681] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_047869d6-7b93-44d0-8cef-79878331cc42 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_047869d6-7b93-44d0-8cef-79878331cc42 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage15, context: None 2026-02-24T16:43:34.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.223180] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage15 2026-02-24T16:43:34.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.223200] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:34.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (33 -> 34) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:34.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:34.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3687407406' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:34.936 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3687407406' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:1684 (105): Start update from 34 to 35 (discovery-trial183). 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:2068 (105): Update complete (34 -> 35) (discovery-trial183). 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 34 to 35 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.884083] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_16, nonce 10.20.193.183:0/229109768, rc 0 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.884144] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_16, nonce 10.20.193.183:0/229109768 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_16' nonce='10.20.193.183:0/229109768' 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.892571] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage16, context: None 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.897128] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage16 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:34.897150] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:34.937 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (34 -> 35) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:34.970 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3687407406' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:1684 (132): Start update from 34 to 35 (discovery-trial153). 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:2068 (132): Update complete (34 -> 35) (discovery-trial153). 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 34 to 35 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.887818] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_16, nonce 10.20.193.153:0/1521049753, rc 0 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.888435] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_16, nonce 10.20.193.153:0/1521049753 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_16' nonce='10.20.193.153:0/1521049753' 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.906433] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage16, context: None 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.929858] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage16 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:34.929882] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:34.971 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (34 -> 35) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:35.124 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:1684 (81): Start update from 34 to 35 (discovery-trial151). 2026-02-24T16:43:35.124 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:2068 (81): Update complete (34 -> 35) (discovery-trial151). 2026-02-24T16:43:35.124 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:34338 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage17, no need to remove 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:34] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:34338 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.031125] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_17, nonce 10.20.193.151:0/2388864417, rc 0 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.031188] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_17, nonce 10.20.193.151:0/2388864417 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:1280 (2): Allocated cluster name='cluster_17' nonce='10.20.193.151:0/2388864417' 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.040397] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:2535 (2): Received request to add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage17, context: , client address: IPv4 10.20.193.51:34338 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.045693] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage17 2026-02-24T16:43:35.125 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.045712] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:35.126 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_17, OMAP version: 36 2026-02-24T16:43:35.126 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.126 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.126 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.126498] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:35.126 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.126522] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:35.126 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:35.145 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:35.146 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:35.146 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage18 2026-02-24T16:43:35.146 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage18 --load-balancing-group 3 2026-02-24T16:43:35.179 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:35Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:35.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:1684 (2): Start update from 34 to 35 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:35.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:35.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:35.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:1684 (126): Start update from 34 to 35 (discovery-trial191). 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:2068 (126): Update complete (34 -> 35) (discovery-trial191). 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.891032] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_16, nonce 10.20.193.191:0/160745044, rc 0 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.891093] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_16, nonce 10.20.193.191:0/160745044 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:1280 (2): Allocated cluster name='cluster_16' nonce='10.20.193.191:0/160745044' 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.900419] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO grpc.py:2535 (2): Received request to add bdev_d68a8f73-7e96-4889-b293-122ed5581ea3 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage16, context: None 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.905671] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage16 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:34.905693] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:34] INFO state.py:2068 (2): Update complete (34 -> 35) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-mon[44017]: pgmap v116: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 650 KiB/s rd, 4.1 KiB/s wr, 721 op/s 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO state.py:1684 (132): Start update from 35 to 36 (discovery-trial153). 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO state.py:2068 (132): Update complete (35 -> 36) (discovery-trial153). 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO state.py:1684 (2): Start update from 35 to 36 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:35.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:35.585282] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_17, nonce 10.20.193.153:0/4225534185, rc 0 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:35.585343] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_17, nonce 10.20.193.153:0/4225534185 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO grpc.py:1280 (2): Allocated cluster name='cluster_17' nonce='10.20.193.153:0/4225534185' 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:35.595326] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO grpc.py:2535 (2): Received request to add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage17, context: None 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:35.599978] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage17 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:35.599999] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] INFO state.py:2068 (2): Update complete (35 -> 36) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.834 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-mon[39453]: pgmap v116: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 650 KiB/s rd, 4.1 KiB/s wr, 721 op/s 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO state.py:1684 (81): Start update from 35 to 36 (discovery-trial151). 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO state.py:2068 (81): Update complete (35 -> 36) (discovery-trial151). 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:51970 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage18, no need to remove 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:35.834 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:51970 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.730543] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_18, nonce 10.20.193.151:0/1258203959, rc 0 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.730613] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_18, nonce 10.20.193.151:0/1258203959 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:1280 (2): Allocated cluster name='cluster_18' nonce='10.20.193.151:0/1258203959' 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.739992] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO grpc.py:2535 (2): Received request to add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage18, context: , client address: IPv4 10.20.193.51:51970 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.744389] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage18 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.744409] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_18, OMAP version: 37 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:35] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.836851] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:35.835 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:35.836876] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:35.836 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:35.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO state.py:1684 (2): Start update from 35 to 36 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:35.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:35.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:35.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO state.py:1684 (126): Start update from 35 to 36 (discovery-trial191). 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO state.py:2068 (126): Update complete (35 -> 36) (discovery-trial191). 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:35.589599] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_17, nonce 10.20.193.191:0/416705594, rc 0 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:35.589662] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_17, nonce 10.20.193.191:0/416705594 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO grpc.py:1280 (2): Allocated cluster name='cluster_17' nonce='10.20.193.191:0/416705594' 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:35.599306] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO grpc.py:2535 (2): Received request to add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage17, context: None 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:35.625449] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage17 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:35.625480] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] INFO state.py:2068 (2): Update complete (35 -> 36) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.855 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:35.855 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:35.855 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage19 2026-02-24T16:43:35.855 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage19 --load-balancing-group 4 2026-02-24T16:43:35.888 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:35Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:35.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-mon[43553]: pgmap v116: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 650 KiB/s rd, 4.1 KiB/s wr, 721 op/s 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO state.py:1684 (105): Start update from 35 to 36 (discovery-trial183). 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO state.py:2068 (105): Update complete (35 -> 36) (discovery-trial183). 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO state.py:1684 (2): Start update from 35 to 36 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:35.584812] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_17, nonce 10.20.193.183:0/2375592641, rc 0 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:35.584869] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_17, nonce 10.20.193.183:0/2375592641 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO grpc.py:1280 (2): Allocated cluster name='cluster_17' nonce='10.20.193.183:0/2375592641' 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:35.594364] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO grpc.py:2535 (2): Received request to add bdev_d9d070fa-74fd-4ec2-a7eb-bd8b8d4b2c5d to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage17, context: None 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:35.599083] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage17 2026-02-24T16:43:35.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:35.599103] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:35.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] INFO state.py:2068 (2): Update complete (35 -> 36) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:35.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:35.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.550 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:1684 (81): Start update from 36 to 37 (discovery-trial151). 2026-02-24T16:43:36.551 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:2068 (81): Update complete (36 -> 37) (discovery-trial151). 2026-02-24T16:43:36.551 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:40292 2026-02-24T16:43:36.551 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage19, no need to remove 2026-02-24T16:43:36.551 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:36.551 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:40292 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.448633] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_19, nonce 10.20.193.151:0/272277898, rc 0 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.448696] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_19, nonce 10.20.193.151:0/272277898 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO grpc.py:1280 (2): Allocated cluster name='cluster_19' nonce='10.20.193.151:0/272277898' 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.458687] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO grpc.py:2535 (2): Received request to add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage19, context: , client address: IPv4 10.20.193.51:40292 2026-02-24T16:43:36.552 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.463380] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage19 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.463402] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_19, OMAP version: 38 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.552556] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:36.553 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:36.552583] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:36.554 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:36.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:1684 (132): Start update from 36 to 37 (discovery-trial153). 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:2068 (132): Update complete (36 -> 37) (discovery-trial153). 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 36 to 37 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:36.283833] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_18, nonce 10.20.193.153:0/690674034, rc 0 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:36.283908] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_18, nonce 10.20.193.153:0/690674034 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:1280 (2): Allocated cluster name='cluster_18' nonce='10.20.193.153:0/690674034' 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:36.293770] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:2535 (2): Received request to add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage18, context: None 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:36.298350] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage18 2026-02-24T16:43:36.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:36.298371] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:36.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:2068 (2): Update complete (36 -> 37) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:36.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.573 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:36.574 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:36.574 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage20 2026-02-24T16:43:36.574 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage20 --load-balancing-group 1 2026-02-24T16:43:36.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:1684 (126): Start update from 36 to 37 (discovery-trial191). 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:2068 (126): Update complete (36 -> 37) (discovery-trial191). 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 36 to 37 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:36.289770] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_18, nonce 10.20.193.191:0/4248736735, rc 0 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:36.289842] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_18, nonce 10.20.193.191:0/4248736735 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:1280 (2): Allocated cluster name='cluster_18' nonce='10.20.193.191:0/4248736735' 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:36.299120] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:36.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:2535 (2): Received request to add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage18, context: None 2026-02-24T16:43:36.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:36.303549] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage18 2026-02-24T16:43:36.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:36.303574] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:36.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:2068 (2): Update complete (36 -> 37) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:36.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.611 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:36Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:36.680 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:1684 (105): Start update from 36 to 37 (discovery-trial183). 2026-02-24T16:43:36.680 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:2068 (105): Update complete (36 -> 37) (discovery-trial183). 2026-02-24T16:43:36.680 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 36 to 37 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:36.681 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:36.681 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:36.681 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:36.282889] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_18, nonce 10.20.193.183:0/1878777718, rc 0 2026-02-24T16:43:36.681 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:36.283237] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_18, nonce 10.20.193.183:0/1878777718 2026-02-24T16:43:36.681 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:1280 (2): Allocated cluster name='cluster_18' nonce='10.20.193.183:0/1878777718' 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:36.294028] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:2535 (2): Received request to add bdev_c883aa1e-10be-4039-b95e-fc44869c1f47 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage18, context: None 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:36.298403] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage18 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:36.298421] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:2068 (2): Update complete (36 -> 37) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.682 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:36.971 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1477444417' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:36.972 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3067381588' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:36.974 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1477444417' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:36.974 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3067381588' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:36.974 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1477444417' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:36.974 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3067381588' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:1684 (105): Start update from 37 to 38 (discovery-trial183). 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:2068 (105): Update complete (37 -> 38) (discovery-trial183). 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 37 to 38 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.002015] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_19, nonce 10.20.193.183:0/2385577885, rc 0 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.002080] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_19, nonce 10.20.193.183:0/2385577885 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_19' nonce='10.20.193.183:0/2385577885' 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.011795] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage19, context: None 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.016376] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage19 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.016397] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (37 -> 38) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:1684 (81): Start update from 37 to 38 (discovery-trial151). 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:36] INFO state.py:2068 (81): Update complete (37 -> 38) (discovery-trial151). 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:46632 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage20, no need to remove 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:46632 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.164679] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_20, nonce 10.20.193.151:0/3371689380, rc 0 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.164758] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_20, nonce 10.20.193.151:0/3371689380 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_20' nonce='10.20.193.151:0/3371689380' 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.175149] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage20, context: , client address: IPv4 10.20.193.51:46632 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.179690] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage20 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.179710] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_20, OMAP version: 39 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.275 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:43:37.297 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:37.297 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:37.297 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:43:37.297 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 20 2026-02-24T16:43:37.298 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:37.298 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage21 2026-02-24T16:43:37.298 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage21 --load-balancing-group 2 2026-02-24T16:43:37.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:1684 (132): Start update from 37 to 38 (discovery-trial153). 2026-02-24T16:43:37.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:2068 (132): Update complete (37 -> 38) (discovery-trial153). 2026-02-24T16:43:37.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 37 to 38 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.001967] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_19, nonce 10.20.193.153:0/405497948, rc 0 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.002030] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_19, nonce 10.20.193.153:0/405497948 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_19' nonce='10.20.193.153:0/405497948' 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.011555] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage19, context: None 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.016077] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage19 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.016099] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (37 -> 38) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:37.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.330 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:37Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:37.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:1684 (126): Start update from 37 to 38 (discovery-trial191). 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:2068 (126): Update complete (37 -> 38) (discovery-trial191). 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO state.py:1684 (2): Start update from 37 to 38 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:36] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.007982] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_19, nonce 10.20.193.191:0/1164001607, rc 0 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.008045] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_19, nonce 10.20.193.191:0/1164001607 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_19' nonce='10.20.193.191:0/1164001607' 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.018694] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_5980b9b8-796e-49f0-a65d-cd0f3161347f to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage19, context: None 2026-02-24T16:43:37.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.023207] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage19 2026-02-24T16:43:37.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.023230] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:37.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (37 -> 38) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:37.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.680 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.275530] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:37.680 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.275556] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-mon[43553]: pgmap v117: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 961 KiB/s rd, 5.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO state.py:1684 (2): Start update from 38 to 39 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO state.py:1684 (105): Start update from 38 to 39 (discovery-trial183). 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO state.py:2068 (105): Update complete (38 -> 39) (discovery-trial183). 2026-02-24T16:43:37.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.727950] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_20, nonce 10.20.193.183:0/191496242, rc 0 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.728019] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_20, nonce 10.20.193.183:0/191496242 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_20' nonce='10.20.193.183:0/191496242' 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.737487] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage20, context: None 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.742027] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage20 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:37.742052] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:37.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (38 -> 39) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:37.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:37] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-mon[39453]: pgmap v117: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 961 KiB/s rd, 5.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:37.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:1684 (81): Start update from 38 to 39 (discovery-trial151). 2026-02-24T16:43:37.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:2068 (81): Update complete (38 -> 39) (discovery-trial151). 2026-02-24T16:43:37.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:52666 2026-02-24T16:43:37.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage21, no need to remove 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_33263a91-b0f5-4879-8060-11a137a1edea from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:52666 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.891489] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_21, nonce 10.20.193.151:0/3101374276, rc 0 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.891546] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_21, nonce 10.20.193.151:0/3101374276 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_21' nonce='10.20.193.151:0/3101374276' 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.900548] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_33263a91-b0f5-4879-8060-11a137a1edea rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_33263a91-b0f5-4879-8060-11a137a1edea to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage21, context: , client address: IPv4 10.20.193.51:52666 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.905346] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage21 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:37.905366] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:37.988 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_1, OMAP version: 40 2026-02-24T16:43:37.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:37.989 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:37] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.001 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:38.028 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:38.028 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:38.028 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage22 2026-02-24T16:43:38.029 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage22 --load-balancing-group 3 2026-02-24T16:43:38.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-mon[44017]: pgmap v117: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 961 KiB/s rd, 5.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO state.py:1684 (132): Start update from 38 to 39 (discovery-trial153). 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO state.py:2068 (132): Update complete (38 -> 39) (discovery-trial153). 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO state.py:1684 (2): Start update from 38 to 39 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.723715] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_20, nonce 10.20.193.153:0/1944161772, rc 0 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.723775] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_20, nonce 10.20.193.153:0/1944161772 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_20' nonce='10.20.193.153:0/1944161772' 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.733507] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage20, context: None 2026-02-24T16:43:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.738017] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage20 2026-02-24T16:43:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:37.738039] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (38 -> 39) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.065 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:38Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:38.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO state.py:1684 (126): Start update from 38 to 39 (discovery-trial191). 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO state.py:2068 (126): Update complete (38 -> 39) (discovery-trial191). 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO state.py:1684 (2): Start update from 38 to 39 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.735546] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_20, nonce 10.20.193.191:0/4007670995, rc 0 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.735611] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_20, nonce 10.20.193.191:0/4007670995 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:1280 (2): Allocated cluster name='cluster_20' nonce='10.20.193.191:0/4007670995' 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.744649] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO grpc.py:2535 (2): Received request to add bdev_e3fd8e09-33ec-4b92-9142-92798b8a4431 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage20, context: None 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.749135] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage20 2026-02-24T16:43:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:37.749157] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:38.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] INFO state.py:2068 (2): Update complete (38 -> 39) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:38.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.415 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.001111] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:38.416 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.001140] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:38.715 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO state.py:1684 (81): Start update from 39 to 40 (discovery-trial151). 2026-02-24T16:43:38.715 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO state.py:2068 (81): Update complete (39 -> 40) (discovery-trial151). 2026-02-24T16:43:38.715 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:33438 2026-02-24T16:43:38.715 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage22, no need to remove 2026-02-24T16:43:38.715 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:33438 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.620961] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_22, nonce 10.20.193.151:0/2886905364, rc 0 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.621024] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_22, nonce 10.20.193.151:0/2886905364 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO grpc.py:1280 (2): Allocated cluster name='cluster_22' nonce='10.20.193.151:0/2886905364' 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.630175] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO grpc.py:2535 (2): Received request to add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage22, context: , client address: IPv4 10.20.193.51:33438 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.634851] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage22 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.634871] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_2, OMAP version: 41 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:38] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.718300] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:38.716 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:38.718327] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:38.718 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO state.py:1684 (2): Start update from 39 to 40 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO state.py:1684 (105): Start update from 39 to 40 (discovery-trial183). 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO state.py:2068 (105): Update complete (39 -> 40) (discovery-trial183). 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_33263a91-b0f5-4879-8060-11a137a1edea from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:38.445810] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_21, nonce 10.20.193.183:0/2297985623, rc 0 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:38.446016] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_21, nonce 10.20.193.183:0/2297985623 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO grpc.py:1280 (2): Allocated cluster name='cluster_21' nonce='10.20.193.183:0/2297985623' 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:38.455433] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_33263a91-b0f5-4879-8060-11a137a1edea rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO grpc.py:2535 (2): Received request to add bdev_33263a91-b0f5-4879-8060-11a137a1edea to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage21, context: None 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:38.460103] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage21 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:38.460124] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] INFO state.py:2068 (2): Update complete (39 -> 40) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:38] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.738 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:38.738 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:38.738 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage23 2026-02-24T16:43:38.738 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage23 --load-balancing-group 4 2026-02-24T16:43:38.778 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:38Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:38.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO state.py:1684 (132): Start update from 39 to 40 (discovery-trial153). 2026-02-24T16:43:38.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO state.py:1684 (2): Start update from 39 to 40 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:38.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO state.py:2068 (132): Update complete (39 -> 40) (discovery-trial153). 2026-02-24T16:43:38.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_33263a91-b0f5-4879-8060-11a137a1edea from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:38.441742] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_21, nonce 10.20.193.153:0/2746088831, rc 0 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:38.451444] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_21, nonce 10.20.193.153:0/2746088831 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO grpc.py:1280 (2): Allocated cluster name='cluster_21' nonce='10.20.193.153:0/2746088831' 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:38.460234] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_33263a91-b0f5-4879-8060-11a137a1edea rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO grpc.py:2535 (2): Received request to add bdev_33263a91-b0f5-4879-8060-11a137a1edea to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage21, context: None 2026-02-24T16:43:38.816 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:38.464747] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage21 2026-02-24T16:43:38.816 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:38.464769] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:38.816 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] INFO state.py:2068 (2): Update complete (39 -> 40) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:38.816 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.816 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:38] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO state.py:1684 (126): Start update from 39 to 40 (discovery-trial191). 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO state.py:2068 (126): Update complete (39 -> 40) (discovery-trial191). 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO state.py:1684 (2): Start update from 39 to 40 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_33263a91-b0f5-4879-8060-11a137a1edea from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:38.449347] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_21, nonce 10.20.193.191:0/1561944734, rc 0 2026-02-24T16:43:38.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:38.449409] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_21, nonce 10.20.193.191:0/1561944734 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO grpc.py:1280 (2): Allocated cluster name='cluster_21' nonce='10.20.193.191:0/1561944734' 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:38.458919] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_33263a91-b0f5-4879-8060-11a137a1edea rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO grpc.py:2535 (2): Received request to add bdev_33263a91-b0f5-4879-8060-11a137a1edea to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage21, context: None 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:38.464330] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage21 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:38.464350] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] INFO state.py:2068 (2): Update complete (39 -> 40) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:38.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:38 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:38] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.227 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:38 trial153 ceph-mon[44017]: pgmap v118: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 932 KiB/s rd, 6.0 KiB/s wr, 1.00k op/s 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:1684 (132): Start update from 40 to 41 (discovery-trial153). 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:2068 (132): Update complete (40 -> 41) (discovery-trial153). 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 40 to 41 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.172316] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_22, nonce 10.20.193.153:0/611079539, rc 0 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.172390] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_22, nonce 10.20.193.153:0/611079539 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_22' nonce='10.20.193.153:0/611079539' 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.181351] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage22, context: None 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.185906] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage22 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.185927] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:39.228 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (40 -> 41) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:39.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:38 trial183 ceph-mon[43553]: pgmap v118: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 932 KiB/s rd, 6.0 KiB/s wr, 1.00k op/s 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:1684 (105): Start update from 40 to 41 (discovery-trial183). 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:2068 (105): Update complete (40 -> 41) (discovery-trial183). 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 40 to 41 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.167957] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_22, nonce 10.20.193.183:0/2448778133, rc 0 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.168243] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_22, nonce 10.20.193.183:0/2448778133 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_22' nonce='10.20.193.183:0/2448778133' 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.187247] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage22, context: None 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.199404] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage22 2026-02-24T16:43:39.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.199433] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:39.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:38 trial151 ceph-mon[39453]: pgmap v118: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 932 KiB/s rd, 6.0 KiB/s wr, 1.00k op/s 2026-02-24T16:43:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:1684 (81): Start update from 40 to 41 (discovery-trial151). 2026-02-24T16:43:39.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:2068 (81): Update complete (40 -> 41) (discovery-trial151). 2026-02-24T16:43:39.442 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:39.466 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:39.466 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:39.466 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage24 2026-02-24T16:43:39.466 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage24 --load-balancing-group 1 2026-02-24T16:43:39.505 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:39.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:1684 (126): Start update from 40 to 41 (discovery-trial191). 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:2068 (126): Update complete (40 -> 41) (discovery-trial191). 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 40 to 41 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:39.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.179816] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_22, nonce 10.20.193.191:0/4190362484, rc 0 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.179868] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_22, nonce 10.20.193.191:0/4190362484 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_22' nonce='10.20.193.191:0/4190362484' 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.190049] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_d78cb7f3-762d-40e2-abb5-ef80b2267e84 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage22, context: None 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.194745] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage22 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.194766] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (40 -> 41) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (40 -> 41) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:39.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:40722 2026-02-24T16:43:39.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage23, no need to remove 2026-02-24T16:43:39.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:39.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:40722 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.334881] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_23, nonce 10.20.193.151:0/1356786071, rc 0 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.335623] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_23, nonce 10.20.193.151:0/1356786071 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_23' nonce='10.20.193.151:0/1356786071' 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.345622] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage23, context: , client address: IPv4 10.20.193.51:40722 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.350014] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage23 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.350034] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_3, OMAP version: 42 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:39.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.441597] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:39.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:39.441627] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:40.157 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3175901847' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:1684 (81): Start update from 41 to 42 (discovery-trial151). 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:2068 (81): Update complete (41 -> 42) (discovery-trial151). 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:56880 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage24, no need to remove 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:39] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:56880 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.053664] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_24, nonce 10.20.193.151:0/404244213, rc 0 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.053729] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_24, nonce 10.20.193.151:0/404244213 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:1280 (2): Allocated cluster name='cluster_24' nonce='10.20.193.151:0/404244213' 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.062722] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:2535 (2): Received request to add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage24, context: , client address: IPv4 10.20.193.51:56880 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.067208] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage24 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.067228] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_4, OMAP version: 43 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.160537] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:40.158 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.160566] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:40.159 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:40.180 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:40.180 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:40.180 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage25 2026-02-24T16:43:40.180 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage25 --load-balancing-group 2 2026-02-24T16:43:40.210 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:40Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:40.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3175901847' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:40.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 41 to 42 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:40.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:1684 (105): Start update from 41 to 42 (discovery-trial183). 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:2068 (105): Update complete (41 -> 42) (discovery-trial183). 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.889794] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_23, nonce 10.20.193.183:0/195741343, rc 0 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.890253] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_23, nonce 10.20.193.183:0/195741343 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_23' nonce='10.20.193.183:0/195741343' 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.900172] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage23, context: None 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.904558] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage23 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:39.904579] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:39 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (41 -> 42) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3175901847' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:1684 (132): Start update from 41 to 42 (discovery-trial153). 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:2068 (132): Update complete (41 -> 42) (discovery-trial153). 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 41 to 42 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.889468] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_23, nonce 10.20.193.153:0/436510800, rc 0 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.889527] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_23, nonce 10.20.193.153:0/436510800 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_23' nonce='10.20.193.153:0/436510800' 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.899522] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage23, context: None 2026-02-24T16:43:40.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.904076] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage23 2026-02-24T16:43:40.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:39.904098] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:40.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:39 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (41 -> 42) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:40.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:1684 (126): Start update from 41 to 42 (discovery-trial191). 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:2068 (126): Update complete (41 -> 42) (discovery-trial191). 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:1684 (2): Start update from 41 to 42 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.894892] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_23, nonce 10.20.193.191:0/3884312194, rc 0 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.894945] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_23, nonce 10.20.193.191:0/3884312194 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:1280 (2): Allocated cluster name='cluster_23' nonce='10.20.193.191:0/3884312194' 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.905933] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO grpc.py:2535 (2): Received request to add bdev_697843d5-c53e-4486-a787-6dbdf2e25dcc to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage23, context: None 2026-02-24T16:43:40.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.910455] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage23 2026-02-24T16:43:40.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:39.910476] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:40.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:39 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:39] INFO state.py:2068 (2): Update complete (41 -> 42) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:40.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO state.py:1684 (126): Start update from 42 to 43 (discovery-trial191). 2026-02-24T16:43:40.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO state.py:2068 (126): Update complete (42 -> 43) (discovery-trial191). 2026-02-24T16:43:40.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO state.py:1684 (2): Start update from 42 to 43 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:40.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:40.610726] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_24, nonce 10.20.193.191:0/3364425446, rc 0 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:40.611302] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_24, nonce 10.20.193.191:0/3364425446 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO grpc.py:1280 (2): Allocated cluster name='cluster_24' nonce='10.20.193.191:0/3364425446' 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:40.629310] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO grpc.py:2535 (2): Received request to add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage24, context: None 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:40.638506] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage24 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:40.638537] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] INFO state.py:2068 (2): Update complete (42 -> 43) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:40 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO state.py:1684 (81): Start update from 42 to 43 (discovery-trial151). 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO state.py:2068 (81): Update complete (42 -> 43) (discovery-trial151). 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:57722 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage25, no need to remove 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:57722 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.794683] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_25, nonce 10.20.193.151:0/1646534519, rc 0 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.794753] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_25, nonce 10.20.193.151:0/1646534519 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:1280 (2): Allocated cluster name='cluster_25' nonce='10.20.193.151:0/1646534519' 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.802840] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.893 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO grpc.py:2535 (2): Received request to add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage25, context: , client address: IPv4 10.20.193.51:57722 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.807480] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage25 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.807500] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_5, OMAP version: 44 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:40] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.895920] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:40.894 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:40.895948] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:40.895 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:40.917 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:40.917 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:40.917 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage26 2026-02-24T16:43:40.917 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage26 --load-balancing-group 3 2026-02-24T16:43:40.958 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:40Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO state.py:1684 (132): Start update from 42 to 43 (discovery-trial153). 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO state.py:2068 (132): Update complete (42 -> 43) (discovery-trial153). 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO state.py:1684 (2): Start update from 42 to 43 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:40.603856] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_24, nonce 10.20.193.153:0/2477134002, rc 0 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:40.604425] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_24, nonce 10.20.193.153:0/2477134002 2026-02-24T16:43:40.980 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO grpc.py:1280 (2): Allocated cluster name='cluster_24' nonce='10.20.193.153:0/2477134002' 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:40.612973] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO grpc.py:2535 (2): Received request to add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage24, context: None 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:40.617654] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage24 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:40.617675] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] INFO state.py:2068 (2): Update complete (42 -> 43) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.981 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO state.py:1684 (2): Start update from 42 to 43 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO state.py:1684 (105): Start update from 42 to 43 (discovery-trial183). 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO state.py:2068 (105): Update complete (42 -> 43) (discovery-trial183). 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:40.604596] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_24, nonce 10.20.193.183:0/1413723500, rc 0 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:40.605252] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_24, nonce 10.20.193.183:0/1413723500 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO grpc.py:1280 (2): Allocated cluster name='cluster_24' nonce='10.20.193.183:0/1413723500' 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:40.616247] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:40.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO grpc.py:2535 (2): Received request to add bdev_bf2e4988-73d0-431b-82a3-e4b3011875b2 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage24, context: None 2026-02-24T16:43:40.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:40.620875] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage24 2026-02-24T16:43:40.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:40.620897] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:40.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] INFO state.py:2068 (2): Update complete (42 -> 43) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:40.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:40.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:40] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:40 trial183 ceph-mon[43553]: pgmap v119: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 947 KiB/s rd, 5.0 KiB/s wr, 1.04k op/s 2026-02-24T16:43:41.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:40 trial151 ceph-mon[39453]: pgmap v119: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 947 KiB/s rd, 5.0 KiB/s wr, 1.04k op/s 2026-02-24T16:43:41.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:40 trial153 ceph-mon[44017]: pgmap v119: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 947 KiB/s rd, 5.0 KiB/s wr, 1.04k op/s 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO state.py:1684 (126): Start update from 43 to 44 (discovery-trial191). 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO state.py:2068 (126): Update complete (43 -> 44) (discovery-trial191). 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO state.py:1684 (2): Start update from 43 to 44 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:41.365858] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_25, nonce 10.20.193.191:0/3670331793, rc 0 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:41.365923] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_25, nonce 10.20.193.191:0/3670331793 2026-02-24T16:43:41.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO grpc.py:1280 (2): Allocated cluster name='cluster_25' nonce='10.20.193.191:0/3670331793' 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:41.375534] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO grpc.py:2535 (2): Received request to add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage25, context: None 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:41.380156] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage25 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:41.380176] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] INFO state.py:2068 (2): Update complete (43 -> 44) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:41 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:41] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.608 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO state.py:1684 (81): Start update from 43 to 44 (discovery-trial151). 2026-02-24T16:43:41.608 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO state.py:2068 (81): Update complete (43 -> 44) (discovery-trial151). 2026-02-24T16:43:41.608 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:44668 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage26, no need to remove 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:44668 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.507912] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_26, nonce 10.20.193.151:0/2902886269, rc 0 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.507980] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_26, nonce 10.20.193.151:0/2902886269 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO grpc.py:1280 (2): Allocated cluster name='cluster_26' nonce='10.20.193.151:0/2902886269' 2026-02-24T16:43:41.609 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.517433] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO grpc.py:2535 (2): Received request to add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage26, context: , client address: IPv4 10.20.193.51:44668 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.522066] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage26 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.522087] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_6, OMAP version: 45 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:41] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.610370] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:41.610 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:41 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:41.610396] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:41.611 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:41.637 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:41.638 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:41.638 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage27 2026-02-24T16:43:41.638 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage27 --load-balancing-group 4 2026-02-24T16:43:41.679 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:41Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:41.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO state.py:1684 (2): Start update from 43 to 44 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO state.py:1684 (105): Start update from 43 to 44 (discovery-trial183). 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO state.py:2068 (105): Update complete (43 -> 44) (discovery-trial183). 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:41.357401] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_25, nonce 10.20.193.183:0/3897643445, rc 0 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:41.357486] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_25, nonce 10.20.193.183:0/3897643445 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO grpc.py:1280 (2): Allocated cluster name='cluster_25' nonce='10.20.193.183:0/3897643445' 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:41.366251] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:41.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO grpc.py:2535 (2): Received request to add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage25, context: None 2026-02-24T16:43:41.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:41.371051] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage25 2026-02-24T16:43:41.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:41.371071] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:41.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] INFO state.py:2068 (2): Update complete (43 -> 44) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:41.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:41 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:41] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO state.py:1684 (132): Start update from 43 to 44 (discovery-trial153). 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO state.py:2068 (132): Update complete (43 -> 44) (discovery-trial153). 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO state.py:1684 (2): Start update from 43 to 44 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:41.361726] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_25, nonce 10.20.193.153:0/3339093121, rc 0 2026-02-24T16:43:41.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:41.361818] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_25, nonce 10.20.193.153:0/3339093121 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO grpc.py:1280 (2): Allocated cluster name='cluster_25' nonce='10.20.193.153:0/3339093121' 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:41.370977] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO grpc.py:2535 (2): Received request to add bdev_d59628d5-17f8-4bf7-a126-95d793d0a138 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage25, context: None 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:41.375546] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage25 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:41.375568] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] INFO state.py:2068 (2): Update complete (43 -> 44) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:41.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:41 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:41] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 44 to 45 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:42.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:1684 (132): Start update from 44 to 45 (discovery-trial153). 2026-02-24T16:43:42.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:42.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:2068 (132): Update complete (44 -> 45) (discovery-trial153). 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.059061] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_26, nonce 10.20.193.153:0/3090943399, rc 0 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.059121] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_26, nonce 10.20.193.153:0/3090943399 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_26' nonce='10.20.193.153:0/3090943399' 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.068804] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage26, context: None 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.073540] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage26 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.073562] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (44 -> 45) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:1684 (126): Start update from 44 to 45 (discovery-trial191). 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:2068 (126): Update complete (44 -> 45) (discovery-trial191). 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 44 to 45 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.068014] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_26, nonce 10.20.193.191:0/1161466297, rc 0 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.068070] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_26, nonce 10.20.193.191:0/1161466297 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_26' nonce='10.20.193.191:0/1161466297' 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.078226] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage26, context: None 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.082800] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage26 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.082823] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (44 -> 45) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:1684 (81): Start update from 44 to 45 (discovery-trial151). 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:2068 (81): Update complete (44 -> 45) (discovery-trial151). 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:47664 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage27, no need to remove 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4919133d-7d04-4f44-8d58-32e340ab9034 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:47664 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.258285] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_27, nonce 10.20.193.151:0/2468469939, rc 0 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.258622] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_27, nonce 10.20.193.151:0/2468469939 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_27' nonce='10.20.193.151:0/2468469939' 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.268615] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage27, context: , client address: IPv4 10.20.193.51:47664 2026-02-24T16:43:42.367 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.273328] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage27 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.273347] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_7, OMAP version: 46 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.369969] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:42.368 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.369992] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:42.369 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:42.391 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:42.391 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:42.391 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage28 2026-02-24T16:43:42.392 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage28 --load-balancing-group 1 2026-02-24T16:43:42.426 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:42Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:42.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 44 to 45 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:1684 (105): Start update from 44 to 45 (discovery-trial183). 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:2068 (105): Update complete (44 -> 45) (discovery-trial183). 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.062374] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_26, nonce 10.20.193.183:0/3088893722, rc 0 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.063252] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_26, nonce 10.20.193.183:0/3088893722 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_26' nonce='10.20.193.183:0/3088893722' 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.073239] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_85b21803-9ff2-4dc2-bebf-ba6ba79bb040 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage26, context: None 2026-02-24T16:43:42.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.077861] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage26 2026-02-24T16:43:42.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.077883] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:42.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (44 -> 45) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:42.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:42.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1086531873' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:43.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-mon[44017]: pgmap v120: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 7.0 KiB/s wr, 1.41k op/s 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:1684 (132): Start update from 45 to 46 (discovery-trial153). 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:2068 (132): Update complete (45 -> 46) (discovery-trial153). 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 45 to 46 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4919133d-7d04-4f44-8d58-32e340ab9034 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.809276] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_27, nonce 10.20.193.153:0/129718721, rc 0 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.809436] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_27, nonce 10.20.193.153:0/129718721 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_27' nonce='10.20.193.153:0/129718721' 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.825422] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage27, context: None 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.831636] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage27 2026-02-24T16:43:43.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:42.831656] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:43.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (45 -> 46) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:43.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:42 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.067 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1086531873' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:43.067 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-mon[39453]: pgmap v120: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 7.0 KiB/s wr, 1.41k op/s 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:1684 (81): Start update from 45 to 46 (discovery-trial151). 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:2068 (81): Update complete (45 -> 46) (discovery-trial151). 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:43330 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage28, no need to remove 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:43330 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.966221] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_28, nonce 10.20.193.151:0/2691574859, rc 0 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.966280] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_28, nonce 10.20.193.151:0/2691574859 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_28' nonce='10.20.193.151:0/2691574859' 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.974021] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage28, context: , client address: IPv4 10.20.193.51:43330 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.978451] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage28 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:42.978470] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_8, OMAP version: 47 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:42 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.070303] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:43.067 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.070329] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:43.070 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:43.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 45 to 46 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:43.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4919133d-7d04-4f44-8d58-32e340ab9034 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:1684 (126): Start update from 45 to 46 (discovery-trial191). 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:2068 (126): Update complete (45 -> 46) (discovery-trial191). 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.818831] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_27, nonce 10.20.193.191:0/3204369234, rc 0 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.818894] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_27, nonce 10.20.193.191:0/3204369234 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_27' nonce='10.20.193.191:0/3204369234' 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.828328] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage27, context: None 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.832980] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage27 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:42.833000] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (45 -> 46) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:42 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:42] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.091 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:43.091 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:43.091 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage29 2026-02-24T16:43:43.091 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage29 --load-balancing-group 2 2026-02-24T16:43:43.132 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:43Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:43.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1086531873' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:43.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-mon[43553]: pgmap v120: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 7.0 KiB/s wr, 1.41k op/s 2026-02-24T16:43:43.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:1684 (2): Start update from 45 to 46 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:43.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4919133d-7d04-4f44-8d58-32e340ab9034 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:1684 (105): Start update from 45 to 46 (discovery-trial183). 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:2068 (105): Update complete (45 -> 46) (discovery-trial183). 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.813463] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_27, nonce 10.20.193.183:0/535213684, rc 0 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.813521] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_27, nonce 10.20.193.183:0/535213684 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:1280 (2): Allocated cluster name='cluster_27' nonce='10.20.193.183:0/535213684' 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.822632] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO grpc.py:2535 (2): Received request to add bdev_4919133d-7d04-4f44-8d58-32e340ab9034 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage27, context: None 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.827253] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage27 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:42.827276] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] INFO state.py:2068 (2): Update complete (45 -> 46) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:42 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:42] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO state.py:1684 (132): Start update from 46 to 47 (discovery-trial153). 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO state.py:2068 (132): Update complete (46 -> 47) (discovery-trial153). 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO state.py:1684 (2): Start update from 46 to 47 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:43.517644] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_28, nonce 10.20.193.153:0/1500451886, rc 0 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:43.517704] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_28, nonce 10.20.193.153:0/1500451886 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO grpc.py:1280 (2): Allocated cluster name='cluster_28' nonce='10.20.193.153:0/1500451886' 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:43.526964] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO grpc.py:2535 (2): Received request to add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage28, context: None 2026-02-24T16:43:43.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:43.531455] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage28 2026-02-24T16:43:43.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:43.531475] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:43.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] INFO state.py:2068 (2): Update complete (46 -> 47) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:43.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:43 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:43] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.814 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO state.py:1684 (81): Start update from 46 to 47 (discovery-trial151). 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO state.py:2068 (81): Update complete (46 -> 47) (discovery-trial151). 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:46856 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage29, no need to remove 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:46856 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.714291] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_29, nonce 10.20.193.151:0/5059263, rc 0 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.714353] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_29, nonce 10.20.193.151:0/5059263 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO grpc.py:1280 (2): Allocated cluster name='cluster_29' nonce='10.20.193.151:0/5059263' 2026-02-24T16:43:43.815 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.724003] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO grpc.py:2535 (2): Received request to add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage29, context: , client address: IPv4 10.20.193.51:46856 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.728509] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage29 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.728528] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_9, OMAP version: 48 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:43] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.817005] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:43.816 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:43 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:43.817029] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:43.817 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:43.839 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:43.839 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:43.839 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage30 2026-02-24T16:43:43.839 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage30 --load-balancing-group 3 2026-02-24T16:43:43.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO state.py:1684 (2): Start update from 46 to 47 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO state.py:1684 (126): Start update from 46 to 47 (discovery-trial191). 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO state.py:2068 (126): Update complete (46 -> 47) (discovery-trial191). 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:43.521206] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_28, nonce 10.20.193.191:0/1104941720, rc 0 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:43.521265] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_28, nonce 10.20.193.191:0/1104941720 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO grpc.py:1280 (2): Allocated cluster name='cluster_28' nonce='10.20.193.191:0/1104941720' 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:43.530373] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO grpc.py:2535 (2): Received request to add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage28, context: None 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:43.534910] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage28 2026-02-24T16:43:43.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:43.534932] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:43.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] INFO state.py:2068 (2): Update complete (46 -> 47) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:43.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:43 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:43] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.878 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:43Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:43.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO state.py:1684 (2): Start update from 46 to 47 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO state.py:1684 (105): Start update from 46 to 47 (discovery-trial183). 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO state.py:2068 (105): Update complete (46 -> 47) (discovery-trial183). 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:43.519005] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_28, nonce 10.20.193.183:0/1695796960, rc 0 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:43.519066] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_28, nonce 10.20.193.183:0/1695796960 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO grpc.py:1280 (2): Allocated cluster name='cluster_28' nonce='10.20.193.183:0/1695796960' 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:43.528793] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:43.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO grpc.py:2535 (2): Received request to add bdev_47256c90-d0c1-47e8-822d-cea30bc74f24 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage28, context: None 2026-02-24T16:43:43.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:43.533293] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage28 2026-02-24T16:43:43.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:43.533313] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:43.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] INFO state.py:2068 (2): Update complete (46 -> 47) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:43.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:43.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:43] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.510 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:1684 (81): Start update from 47 to 48 (discovery-trial151). 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:2068 (81): Update complete (47 -> 48) (discovery-trial151). 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:51986 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage30, no need to remove 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:51986 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.409353] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_30, nonce 10.20.193.151:0/2446557556, rc 0 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.409631] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_30, nonce 10.20.193.151:0/2446557556 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_30' nonce='10.20.193.151:0/2446557556' 2026-02-24T16:43:44.511 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.417951] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage30, context: , client address: IPv4 10.20.193.51:51986 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.422468] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage30 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.422488] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_10, OMAP version: 49 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.512924] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:44.512 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:44.512951] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:44.513 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:44.537 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:44.537 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:44.537 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage31 2026-02-24T16:43:44.537 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage31 --load-balancing-group 4 2026-02-24T16:43:44.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:1684 (132): Start update from 47 to 48 (discovery-trial153). 2026-02-24T16:43:44.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:2068 (132): Update complete (47 -> 48) (discovery-trial153). 2026-02-24T16:43:44.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 47 to 48 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:44.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:44.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.271033] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_29, nonce 10.20.193.153:0/3630829814, rc 0 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.271423] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_29, nonce 10.20.193.153:0/3630829814 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_29' nonce='10.20.193.153:0/3630829814' 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.280431] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage29, context: None 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.300776] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage29 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.300797] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:2068 (2): Update complete (47 -> 48) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.572 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:44Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:44.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 47 to 48 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:1684 (126): Start update from 47 to 48 (discovery-trial191). 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:2068 (126): Update complete (47 -> 48) (discovery-trial191). 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.270760] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_29, nonce 10.20.193.191:0/1572390645, rc 0 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.270815] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_29, nonce 10.20.193.191:0/1572390645 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_29' nonce='10.20.193.191:0/1572390645' 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.280315] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage29, context: None 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.284857] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage29 2026-02-24T16:43:44.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.284878] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:44.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:2068 (2): Update complete (47 -> 48) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:44.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:1684 (105): Start update from 47 to 48 (discovery-trial183). 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:2068 (105): Update complete (47 -> 48) (discovery-trial183). 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 47 to 48 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.265177] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_29, nonce 10.20.193.183:0/2898367009, rc 0 2026-02-24T16:43:44.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.265258] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_29, nonce 10.20.193.183:0/2898367009 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_29' nonce='10.20.193.183:0/2898367009' 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.278249] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_1955cd27-99fd-4931-a07c-9ddaafcfa3f9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage29, context: None 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.287425] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage29 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.287447] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:2068 (2): Update complete (47 -> 48) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:44.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-mon[43553]: pgmap v121: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 940 KiB/s rd, 4.7 KiB/s wr, 1k op/s 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 48 to 49 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:1684 (105): Start update from 48 to 49 (discovery-trial183). 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO state.py:2068 (105): Update complete (48 -> 49) (discovery-trial183). 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.958204] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_30, nonce 10.20.193.183:0/3291883150, rc 0 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.958282] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_30, nonce 10.20.193.183:0/3291883150 2026-02-24T16:43:45.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_30' nonce='10.20.193.183:0/3291883150' 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.967697] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage30, context: None 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.972346] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage30 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:44 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:44.972369] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (48 -> 49) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-mon[39453]: pgmap v121: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 940 KiB/s rd, 4.7 KiB/s wr, 1k op/s 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:1684 (81): Start update from 48 to 49 (discovery-trial151). 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:44 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:44] INFO state.py:2068 (81): Update complete (48 -> 49) (discovery-trial151). 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:59856 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage31, no need to remove 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e58f322b-54d4-498a-9468-0145cb626612 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:59856 2026-02-24T16:43:45.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.134377] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_31, nonce 10.20.193.151:0/3912568528, rc 0 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.134622] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_31, nonce 10.20.193.151:0/3912568528 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:1280 (2): Allocated cluster name='cluster_31' nonce='10.20.193.151:0/3912568528' 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.142911] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e58f322b-54d4-498a-9468-0145cb626612 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:2535 (2): Received request to add bdev_e58f322b-54d4-498a-9468-0145cb626612 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage31, context: , client address: IPv4 10.20.193.51:59856 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.151795] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage31 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.151816] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_11, OMAP version: 50 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.237044] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:45.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.237071] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:45.237 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:45.258 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:45.258 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:45.258 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage32 2026-02-24T16:43:45.258 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage32 --load-balancing-group 1 2026-02-24T16:43:45.298 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:45Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:45.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-mon[44017]: pgmap v121: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 940 KiB/s rd, 4.7 KiB/s wr, 1k op/s 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:1684 (132): Start update from 48 to 49 (discovery-trial153). 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:2068 (132): Update complete (48 -> 49) (discovery-trial153). 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 48 to 49 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.960936] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_30, nonce 10.20.193.153:0/4136285744, rc 0 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.960994] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_30, nonce 10.20.193.153:0/4136285744 2026-02-24T16:43:45.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_30' nonce='10.20.193.153:0/4136285744' 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.971147] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage30, context: None 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.975647] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage30 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:44 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:44.975667] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (48 -> 49) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:1684 (126): Start update from 48 to 49 (discovery-trial191). 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:2068 (126): Update complete (48 -> 49) (discovery-trial191). 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO state.py:1684 (2): Start update from 48 to 49 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.968758] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_30, nonce 10.20.193.191:0/63370744, rc 0 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.968811] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_30, nonce 10.20.193.191:0/63370744 2026-02-24T16:43:45.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:1280 (2): Allocated cluster name='cluster_30' nonce='10.20.193.191:0/63370744' 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.978516] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:44] INFO grpc.py:2535 (2): Received request to add bdev_561ccdee-bf1f-4cee-bfc7-b59a062726db to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage30, context: None 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.983019] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage30 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:44 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:44.983038] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (48 -> 49) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:1684 (81): Start update from 49 to 50 (discovery-trial151). 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:2068 (81): Update complete (49 -> 50) (discovery-trial151). 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:47766 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage32, no need to remove 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:45.931 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_39df5daa-d0fe-46f3-87cd-17f324297665 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:47766 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.821117] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_32, nonce 10.20.193.151:0/1271526557, rc 0 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.821626] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_32, nonce 10.20.193.151:0/1271526557 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:1280 (2): Allocated cluster name='cluster_32' nonce='10.20.193.151:0/1271526557' 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.831550] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO grpc.py:2535 (2): Received request to add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage32, context: , client address: IPv4 10.20.193.51:47766 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.836572] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage32 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.836599] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_12, OMAP version: 51 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.934265] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:45.932 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:45 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:45.934294] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:45.934 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:45.955 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:45.955 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:45.955 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage33 2026-02-24T16:43:45.955 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage33 --load-balancing-group 2 2026-02-24T16:43:45.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO state.py:1684 (2): Start update from 49 to 50 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:45.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO state.py:1684 (105): Start update from 49 to 50 (discovery-trial183). 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO state.py:2068 (105): Update complete (49 -> 50) (discovery-trial183). 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e58f322b-54d4-498a-9468-0145cb626612 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:45.691797] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_31, nonce 10.20.193.183:0/1106279700, rc 0 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:45.691859] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_31, nonce 10.20.193.183:0/1106279700 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO grpc.py:1280 (2): Allocated cluster name='cluster_31' nonce='10.20.193.183:0/1106279700' 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:45.700239] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e58f322b-54d4-498a-9468-0145cb626612 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO grpc.py:2535 (2): Received request to add bdev_e58f322b-54d4-498a-9468-0145cb626612 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage31, context: None 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:45.725392] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage31 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:45.725424] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (49 -> 50) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:45 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:45.997 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:46Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO state.py:1684 (132): Start update from 49 to 50 (discovery-trial153). 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO state.py:2068 (132): Update complete (49 -> 50) (discovery-trial153). 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO state.py:1684 (2): Start update from 49 to 50 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e58f322b-54d4-498a-9468-0145cb626612 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:45.695281] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_31, nonce 10.20.193.153:0/1578338182, rc 0 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:45.695354] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_31, nonce 10.20.193.153:0/1578338182 2026-02-24T16:43:46.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO grpc.py:1280 (2): Allocated cluster name='cluster_31' nonce='10.20.193.153:0/1578338182' 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:45.702866] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e58f322b-54d4-498a-9468-0145cb626612 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO grpc.py:2535 (2): Received request to add bdev_e58f322b-54d4-498a-9468-0145cb626612 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage31, context: None 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:45.707610] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage31 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:45.707631] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (49 -> 50) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:45 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:45] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO state.py:1684 (2): Start update from 49 to 50 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e58f322b-54d4-498a-9468-0145cb626612 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO state.py:1684 (126): Start update from 49 to 50 (discovery-trial191). 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO state.py:2068 (126): Update complete (49 -> 50) (discovery-trial191). 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:45.699133] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_31, nonce 10.20.193.191:0/226188743, rc 0 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:45.699189] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_31, nonce 10.20.193.191:0/226188743 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO grpc.py:1280 (2): Allocated cluster name='cluster_31' nonce='10.20.193.191:0/226188743' 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:45.707998] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e58f322b-54d4-498a-9468-0145cb626612 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO grpc.py:2535 (2): Received request to add bdev_e58f322b-54d4-498a-9468-0145cb626612 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage31, context: None 2026-02-24T16:43:46.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:45.712442] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage31 2026-02-24T16:43:46.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:45.712463] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:46.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] INFO state.py:2068 (2): Update complete (49 -> 50) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:46.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:45 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:45] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO state.py:1684 (81): Start update from 50 to 51 (discovery-trial151). 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO state.py:2068 (81): Update complete (50 -> 51) (discovery-trial151). 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:51484 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage33, no need to remove 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:51484 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:46.530622] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.631 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO grpc.py:2535 (2): Received request to add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage33, context: , client address: IPv4 10.20.193.51:51484 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:46.553810] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage33 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:46.553833] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_13, OMAP version: 52 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:46] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:46.633516] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:46.632 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:46.633544] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:46.633 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:46.652 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:46.652 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:46.652 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage34 2026-02-24T16:43:46.652 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage34 --load-balancing-group 3 2026-02-24T16:43:46.688 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:46Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:46.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO state.py:1684 (105): Start update from 50 to 51 (discovery-trial183). 2026-02-24T16:43:46.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO state.py:2068 (105): Update complete (50 -> 51) (discovery-trial183). 2026-02-24T16:43:46.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO state.py:1684 (2): Start update from 50 to 51 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:46.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_39df5daa-d0fe-46f3-87cd-17f324297665 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:46.377212] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_32, nonce 10.20.193.183:0/1928129541, rc 0 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:46.378249] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_32, nonce 10.20.193.183:0/1928129541 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO grpc.py:1280 (2): Allocated cluster name='cluster_32' nonce='10.20.193.183:0/1928129541' 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:46.387146] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO grpc.py:2535 (2): Received request to add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage32, context: None 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:46.400445] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage32 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:46.400466] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] INFO state.py:2068 (2): Update complete (50 -> 51) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:46] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO state.py:1684 (2): Start update from 50 to 51 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_39df5daa-d0fe-46f3-87cd-17f324297665 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO state.py:1684 (132): Start update from 50 to 51 (discovery-trial153). 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO state.py:2068 (132): Update complete (50 -> 51) (discovery-trial153). 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:46.375139] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_32, nonce 10.20.193.153:0/1444668799, rc 0 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:46.375197] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_32, nonce 10.20.193.153:0/1444668799 2026-02-24T16:43:46.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO grpc.py:1280 (2): Allocated cluster name='cluster_32' nonce='10.20.193.153:0/1444668799' 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:46.384645] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO grpc.py:2535 (2): Received request to add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage32, context: None 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:46.389851] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage32 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:46.389871] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] INFO state.py:2068 (2): Update complete (50 -> 51) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:46] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO state.py:1684 (126): Start update from 50 to 51 (discovery-trial191). 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO state.py:2068 (126): Update complete (50 -> 51) (discovery-trial191). 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO state.py:1684 (2): Start update from 50 to 51 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_39df5daa-d0fe-46f3-87cd-17f324297665 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:46.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:46.384487] bdev_rbd.c:1551:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_32, nonce 10.20.193.191:0/4090729480, rc 0 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:46.384547] bdev_rbd.c:1230:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_32, nonce 10.20.193.191:0/4090729480 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO grpc.py:1280 (2): Allocated cluster name='cluster_32' nonce='10.20.193.191:0/4090729480' 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:46.393814] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO grpc.py:2535 (2): Received request to add bdev_39df5daa-d0fe-46f3-87cd-17f324297665 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage32, context: None 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:46.398455] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage32 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:46.398477] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] INFO state.py:2068 (2): Update complete (50 -> 51) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:46.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:46 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:46] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.311 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2858896719' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.311 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4111512440' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.311 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:46 trial151 ceph-mon[39453]: pgmap v122: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.6 KiB/s wr, 1.30k op/s 2026-02-24T16:43:47.311 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:1684 (81): Start update from 51 to 52 (discovery-trial151). 2026-02-24T16:43:47.311 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:2068 (81): Update complete (51 -> 52) (discovery-trial151). 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:44972 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage34, no need to remove 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:44972 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.219638] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage34, context: , client address: IPv4 10.20.193.51:44972 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.230767] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage34 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.230788] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_14, OMAP version: 53 2026-02-24T16:43:47.312 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.313 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.313 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.313458] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:47.313 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.313489] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:47.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2858896719' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4111512440' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:46 trial153 ceph-mon[44017]: pgmap v122: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.6 KiB/s wr, 1.30k op/s 2026-02-24T16:43:47.314 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:47.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:1684 (132): Start update from 51 to 52 (discovery-trial153). 2026-02-24T16:43:47.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:2068 (132): Update complete (51 -> 52) (discovery-trial153). 2026-02-24T16:43:47.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 51 to 52 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:47.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.090424] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage33, context: None 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.100573] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage33 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.100595] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (51 -> 52) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.337 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:47.337 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:47.337 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage35 2026-02-24T16:43:47.337 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage35 --load-balancing-group 4 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:1684 (126): Start update from 51 to 52 (discovery-trial191). 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 51 to 52 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:2068 (126): Update complete (51 -> 52) (discovery-trial191). 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.094201] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage33, context: None 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.098711] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage33 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.098734] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (51 -> 52) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.382 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:47Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:47.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2858896719' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4111512440' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:47.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:46 trial183 ceph-mon[43553]: pgmap v122: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.6 KiB/s wr, 1.30k op/s 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 51 to 52 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:1684 (105): Start update from 51 to 52 (discovery-trial183). 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:2068 (105): Update complete (51 -> 52) (discovery-trial183). 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.092250] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3aeacf00-aa1c-433c-9d46-b94cdf90b730 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage33, context: None 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.101418] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage33 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.101438] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (51 -> 52) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:47.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.018 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:1684 (81): Start update from 52 to 53 (discovery-trial151). 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:2068 (81): Update complete (52 -> 53) (discovery-trial151). 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:54078 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage35, no need to remove 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:54078 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.925625] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage35, context: , client address: IPv4 10.20.193.51:54078 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.930036] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage35 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:47.930058] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_15, OMAP version: 54 2026-02-24T16:43:48.035 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.036 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:47 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.036 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.021090] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:48.036 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.021117] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:48.036 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:48.043 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:48.043 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:48.043 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage36 2026-02-24T16:43:48.043 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage36 --load-balancing-group 1 2026-02-24T16:43:48.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:1684 (132): Start update from 52 to 53 (discovery-trial153). 2026-02-24T16:43:48.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:2068 (132): Update complete (52 -> 53) (discovery-trial153). 2026-02-24T16:43:48.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 52 to 53 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:48.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.777428] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage34, context: None 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.782641] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage34 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:47.782661] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (52 -> 53) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:47 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.076 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:48Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 52 to 53 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:1684 (126): Start update from 52 to 53 (discovery-trial191). 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:2068 (126): Update complete (52 -> 53) (discovery-trial191). 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.774661] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage34, context: None 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.778972] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage34 2026-02-24T16:43:48.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:47.778996] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:48.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (52 -> 53) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:48.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:47 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:1684 (2): Start update from 52 to 53 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:1684 (105): Start update from 52 to 53 (discovery-trial183). 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:2068 (105): Update complete (52 -> 53) (discovery-trial183). 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.768956] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO grpc.py:2535 (2): Received request to add bdev_3fd1d7af-bc41-4ce5-be3b-421cfd63d96c to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage34, context: None 2026-02-24T16:43:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.773616] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage34 2026-02-24T16:43:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:47.773638] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:43:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] INFO state.py:2068 (2): Update complete (52 -> 53) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:47 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:47] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO state.py:1684 (105): Start update from 53 to 54 (discovery-trial183). 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO state.py:2068 (105): Update complete (53 -> 54) (discovery-trial183). 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO state.py:1684 (2): Start update from 53 to 54 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:48.474248] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO grpc.py:2535 (2): Received request to add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage35, context: None 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:48.496432] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage35 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:48.496465] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:48.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] INFO state.py:2068 (2): Update complete (53 -> 54) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:48.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:48] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO state.py:1684 (81): Start update from 53 to 54 (discovery-trial151). 2026-02-24T16:43:48.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO state.py:2068 (81): Update complete (53 -> 54) (discovery-trial151). 2026-02-24T16:43:48.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:34082 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage36, no need to remove 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d6b120df-b3ac-4704-843a-d114552783c9 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:34082 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.645817] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d6b120df-b3ac-4704-843a-d114552783c9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO grpc.py:2535 (2): Received request to add bdev_d6b120df-b3ac-4704-843a-d114552783c9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage36, context: , client address: IPv4 10.20.193.51:34082 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.650330] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage36 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.650349] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_16, OMAP version: 55 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:48] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.739 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:48.761 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:48.761 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:48.761 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage37 2026-02-24T16:43:48.761 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage37 --load-balancing-group 2 2026-02-24T16:43:48.792 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:48Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO state.py:1684 (132): Start update from 53 to 54 (discovery-trial153). 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO state.py:2068 (132): Update complete (53 -> 54) (discovery-trial153). 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO state.py:1684 (2): Start update from 53 to 54 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:48.469540] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO grpc.py:2535 (2): Received request to add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage35, context: None 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:48.474200] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage35 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:48.474219] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] INFO state.py:2068 (2): Update complete (53 -> 54) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:48] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO state.py:1684 (126): Start update from 53 to 54 (discovery-trial191). 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO state.py:2068 (126): Update complete (53 -> 54) (discovery-trial191). 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO state.py:1684 (2): Start update from 53 to 54 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:48.479284] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:48.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO grpc.py:2535 (2): Received request to add bdev_1c4ecda8-6e7f-4b9c-8a6b-d2f9cd6a8229 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage35, context: None 2026-02-24T16:43:48.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:48.483648] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage35 2026-02-24T16:43:48.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:48.483671] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:43:48.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] INFO state.py:2068 (2): Update complete (53 -> 54) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:48.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:48 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:48] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:48.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.739581] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:48.994 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:48.739610] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:49.245 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1864978395' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:49.246 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:48 trial183 ceph-mon[43553]: pgmap v123: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 6.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:49.246 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:43:49.246 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 54 to 55 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:49.246 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:49.246 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d6b120df-b3ac-4704-843a-d114552783c9 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:49.246 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:1684 (105): Start update from 54 to 55 (discovery-trial183). 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:2068 (105): Update complete (54 -> 55) (discovery-trial183). 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.196444] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d6b120df-b3ac-4704-843a-d114552783c9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_d6b120df-b3ac-4704-843a-d114552783c9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage36, context: None 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.200955] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage36 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.200991] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:49.247 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:2068 (2): Update complete (54 -> 55) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:49.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1864978395' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:49.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-mon[44017]: pgmap v123: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 6.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:49.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:1684 (132): Start update from 54 to 55 (discovery-trial153). 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:2068 (132): Update complete (54 -> 55) (discovery-trial153). 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 54 to 55 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d6b120df-b3ac-4704-843a-d114552783c9 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.191882] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d6b120df-b3ac-4704-843a-d114552783c9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:49.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_d6b120df-b3ac-4704-843a-d114552783c9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage36, context: None 2026-02-24T16:43:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.196354] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage36 2026-02-24T16:43:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.196378] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:2068 (2): Update complete (54 -> 55) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:49.351 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1864978395' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:49.351 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-mon[39453]: pgmap v123: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 6.2 KiB/s wr, 1.10k op/s 2026-02-24T16:43:49.351 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:48 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:43:49.352 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:1684 (81): Start update from 54 to 55 (discovery-trial151). 2026-02-24T16:43:49.352 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:2068 (81): Update complete (54 -> 55) (discovery-trial151). 2026-02-24T16:43:49.352 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:37116 2026-02-24T16:43:49.455 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:1684 (126): Start update from 54 to 55 (discovery-trial191). 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:2068 (126): Update complete (54 -> 55) (discovery-trial191). 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 54 to 55 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d6b120df-b3ac-4704-843a-d114552783c9 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:49.200338] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d6b120df-b3ac-4704-843a-d114552783c9 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_d6b120df-b3ac-4704-843a-d114552783c9 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage36, context: None 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:49.204754] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage36 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:49.204783] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:2068 (2): Update complete (54 -> 55) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:49.456 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.538 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:49.560 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:49.560 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:49.560 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage38 2026-02-24T16:43:49.560 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage38 --load-balancing-group 3 2026-02-24T16:43:49.595 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:49Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:49.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage37, no need to remove 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f2b63888-78d5-427e-986d-35708fc88a7e from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:37116 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:49.443960] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f2b63888-78d5-427e-986d-35708fc88a7e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_f2b63888-78d5-427e-986d-35708fc88a7e to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage37, context: , client address: IPv4 10.20.193.51:37116 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:49.448440] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage37 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:49.448463] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_17, OMAP version: 56 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:49.538865] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:49.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:49.538896] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:49.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:49.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2115586170' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:50.223 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.224 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:1684 (81): Start update from 55 to 56 (discovery-trial151). 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:49] INFO state.py:2068 (81): Update complete (55 -> 56) (discovery-trial151). 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:51972 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage38, no need to remove 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:51972 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.136774] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:2535 (2): Received request to add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage38, context: , client address: IPv4 10.20.193.51:51972 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.141150] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage38 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.141173] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_18, OMAP version: 57 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.224 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.226 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2115586170' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:43:50.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:1684 (105): Start update from 55 to 56 (discovery-trial183). 2026-02-24T16:43:50.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:2068 (105): Update complete (55 -> 56) (discovery-trial183). 2026-02-24T16:43:50.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 55 to 56 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:50.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f2b63888-78d5-427e-986d-35708fc88a7e from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.989399] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f2b63888-78d5-427e-986d-35708fc88a7e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_f2b63888-78d5-427e-986d-35708fc88a7e to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage37, context: None 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.994025] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage37 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:49 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:49.994048] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (55 -> 56) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.247 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:50.247 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:50.247 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage39 2026-02-24T16:43:50.247 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage39 --load-balancing-group 4 2026-02-24T16:43:50.287 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:50Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:50.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2115586170' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:43:50.315 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:43:50.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:1684 (132): Start update from 55 to 56 (discovery-trial153). 2026-02-24T16:43:50.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:2068 (132): Update complete (55 -> 56) (discovery-trial153). 2026-02-24T16:43:50.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 55 to 56 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:50.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:50.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f2b63888-78d5-427e-986d-35708fc88a7e from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.988420] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f2b63888-78d5-427e-986d-35708fc88a7e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_f2b63888-78d5-427e-986d-35708fc88a7e to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage37, context: None 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.997671] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage37 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:49 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:49.997693] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (55 -> 56) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.316 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:1684 (2): Start update from 55 to 56 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:1684 (126): Start update from 55 to 56 (discovery-trial191). 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f2b63888-78d5-427e-986d-35708fc88a7e from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO state.py:2068 (126): Update complete (55 -> 56) (discovery-trial191). 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:49.996760] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f2b63888-78d5-427e-986d-35708fc88a7e rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:49 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:49] INFO grpc.py:2535 (2): Received request to add bdev_f2b63888-78d5-427e-986d-35708fc88a7e to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage37, context: None 2026-02-24T16:43:50.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:50.001193] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage37 2026-02-24T16:43:50.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:50.001218] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:43:50.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (55 -> 56) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:50.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.226577] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.226607] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:50.946 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:1684 (81): Start update from 56 to 57 (discovery-trial151). 2026-02-24T16:43:50.946 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:2068 (81): Update complete (56 -> 57) (discovery-trial151). 2026-02-24T16:43:50.946 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:49916 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage39, no need to remove 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:49916 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.862533] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO grpc.py:2535 (2): Received request to add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage39, context: , client address: IPv4 10.20.193.51:49916 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.866890] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage39 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.866913] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_19, OMAP version: 58 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.949078] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:50.947 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:50.949105] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:50.948 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:50.970 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:50.970 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:50.970 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage40 2026-02-24T16:43:50.970 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage40 --load-balancing-group 1 2026-02-24T16:43:50.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO state.py:1684 (2): Start update from 56 to 57 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:50.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:50.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:50.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO state.py:1684 (105): Start update from 56 to 57 (discovery-trial183). 2026-02-24T16:43:50.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO state.py:2068 (105): Update complete (56 -> 57) (discovery-trial183). 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:50.682615] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO grpc.py:2535 (2): Received request to add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage38, context: None 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:50.687015] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage38 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:50.687036] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (56 -> 57) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:50.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.010 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:51Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO state.py:1684 (132): Start update from 56 to 57 (discovery-trial153). 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO state.py:2068 (132): Update complete (56 -> 57) (discovery-trial153). 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO state.py:1684 (2): Start update from 56 to 57 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:50.688433] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO grpc.py:2535 (2): Received request to add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage38, context: None 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:50.712648] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage38 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:50.712680] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (56 -> 57) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:50 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO state.py:1684 (2): Start update from 56 to 57 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO state.py:1684 (126): Start update from 56 to 57 (discovery-trial191). 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO state.py:2068 (126): Update complete (56 -> 57) (discovery-trial191). 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:50.685270] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO grpc.py:2535 (2): Received request to add bdev_e7481463-7d31-4d0d-84ee-837a0ee299d5 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage38, context: None 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:50.689754] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage38 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:50.689780] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:43:51.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] INFO state.py:2068 (2): Update complete (56 -> 57) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:51.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:50 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:50] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.654 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-mon[39453]: pgmap v124: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 5.3 KiB/s wr, 1.22k op/s 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO state.py:1684 (81): Start update from 57 to 58 (discovery-trial151). 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO state.py:2068 (81): Update complete (57 -> 58) (discovery-trial151). 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:38060 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage40, no need to remove 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:38060 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:51.558992] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO grpc.py:2535 (2): Received request to add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage40, context: , client address: IPv4 10.20.193.51:38060 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:51.563424] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage40 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:51.563445] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_20, OMAP version: 59 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:51] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:51.656968] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:51.654 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:51 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:51.656994] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:51.656 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:43:51.677 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:51.677 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:43:51.677 INFO:tasks.workunit.client.0.trial051.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:43:51.677 INFO:tasks.workunit.client.0.trial051.stderr:++ seq 1 20 2026-02-24T16:43:51.678 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:51.678 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage41 2026-02-24T16:43:51.678 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage41 --load-balancing-group 2 2026-02-24T16:43:51.720 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:51Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:51.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-mon[43553]: pgmap v124: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 5.3 KiB/s wr, 1.22k op/s 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO state.py:1684 (105): Start update from 57 to 58 (discovery-trial183). 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO state.py:2068 (105): Update complete (57 -> 58) (discovery-trial183). 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO state.py:1684 (2): Start update from 57 to 58 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:51.407681] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO grpc.py:2535 (2): Received request to add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage39, context: None 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:51.412101] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage39 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:51.412123] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] INFO state.py:2068 (2): Update complete (57 -> 58) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:51 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:51] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-mon[44017]: pgmap v124: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 5.3 KiB/s wr, 1.22k op/s 2026-02-24T16:43:51.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO state.py:1684 (132): Start update from 57 to 58 (discovery-trial153). 2026-02-24T16:43:51.813 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO state.py:2068 (132): Update complete (57 -> 58) (discovery-trial153). 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO state.py:1684 (2): Start update from 57 to 58 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:51.409906] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO grpc.py:2535 (2): Received request to add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage39, context: None 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:51.414417] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage39 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:51.414441] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] INFO state.py:2068 (2): Update complete (57 -> 58) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:51 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:51] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO state.py:1684 (2): Start update from 57 to 58 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO state.py:1684 (126): Start update from 57 to 58 (discovery-trial191). 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO state.py:2068 (126): Update complete (57 -> 58) (discovery-trial191). 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:51.410005] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO grpc.py:2535 (2): Received request to add bdev_7c5fad51-f463-4927-bfdb-c5c90e7c7645 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage39, context: None 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:51.414397] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage39 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:51.414422] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] INFO state.py:2068 (2): Update complete (57 -> 58) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:51] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 58 to 59 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:1684 (126): Start update from 58 to 59 (discovery-trial191). 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:2068 (126): Update complete (58 -> 59) (discovery-trial191). 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.110639] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage40, context: None 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.115204] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage40 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.115228] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (58 -> 59) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.355 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:1684 (81): Start update from 58 to 59 (discovery-trial151). 2026-02-24T16:43:52.355 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:2068 (81): Update complete (58 -> 59) (discovery-trial151). 2026-02-24T16:43:52.355 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:35470 2026-02-24T16:43:52.355 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage41, no need to remove 2026-02-24T16:43:52.355 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:35470 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.261486] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage41, context: , client address: IPv4 10.20.193.51:35470 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.266055] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage41 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.266081] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_1, OMAP version: 60 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.357634] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:52.356 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.357659] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:52.357 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:52.382 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:52.382 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:52.382 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage42 2026-02-24T16:43:52.382 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage42 --load-balancing-group 3 2026-02-24T16:43:52.421 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:52Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:1684 (105): Start update from 58 to 59 (discovery-trial183). 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:2068 (105): Update complete (58 -> 59) (discovery-trial183). 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 58 to 59 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.103880] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:52.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage40, context: None 2026-02-24T16:43:52.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.108231] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage40 2026-02-24T16:43:52.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.108251] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:52.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (58 -> 59) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:52.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:1684 (132): Start update from 58 to 59 (discovery-trial153). 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:2068 (132): Update complete (58 -> 59) (discovery-trial153). 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 58 to 59 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.102098] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:52.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_95b5e395-9997-489f-8ead-5f6a6bdae442 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage40, context: None 2026-02-24T16:43:52.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.106554] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage40 2026-02-24T16:43:52.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.106577] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:43:52.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (58 -> 59) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:52.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:52.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:1684 (81): Start update from 59 to 60 (discovery-trial151). 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:2068 (81): Update complete (59 -> 60) (discovery-trial151). 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:55820 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage42, no need to remove 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8b96a726-7bf9-4da5-9239-52b473dec480 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:55820 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.977623] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage42, context: , client address: IPv4 10.20.193.51:55820 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.982031] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage42 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:52.982053] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_2, OMAP version: 61 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:52 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.059201] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:53.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.059233] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:53.058 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:1684 (132): Start update from 59 to 60 (discovery-trial153). 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:2068 (132): Update complete (59 -> 60) (discovery-trial153). 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 59 to 60 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.806588] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage41, context: None 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.810989] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage41 2026-02-24T16:43:53.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:52.811009] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:53.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (59 -> 60) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:53.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:52 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:52] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.079 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:53.079 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:53.079 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage43 2026-02-24T16:43:53.079 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage43 --load-balancing-group 4 2026-02-24T16:43:53.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:1684 (126): Start update from 59 to 60 (discovery-trial191). 2026-02-24T16:43:53.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:2068 (126): Update complete (59 -> 60) (discovery-trial191). 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 59 to 60 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.811492] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage41, context: None 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.815860] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage41 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:52.815884] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (59 -> 60) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:52 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:52] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.126 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:53Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:1684 (2): Start update from 59 to 60 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:2774 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:1684 (105): Start update from 59 to 60 (discovery-trial183). 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:2068 (105): Update complete (59 -> 60) (discovery-trial183). 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.808448] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO grpc.py:2535 (2): Received request to add bdev_d5afe5a9-1637-4f3b-b953-fb9578bad99f to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage41, context: None 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.812894] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage41 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:52.812922] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 1 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] INFO state.py:2068 (2): Update complete (59 -> 60) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:52 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:52] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.659 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-mon[39453]: pgmap v125: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.7 KiB/s wr, 1.38k op/s 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO state.py:1684 (81): Start update from 60 to 61 (discovery-trial151). 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO state.py:2068 (81): Update complete (60 -> 61) (discovery-trial151). 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:45050 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage43, no need to remove 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:53.660 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:45050 2026-02-24T16:43:53.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-mon[43553]: pgmap v125: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.7 KiB/s wr, 1.38k op/s 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO state.py:1684 (105): Start update from 60 to 61 (discovery-trial183). 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO state.py:2068 (105): Update complete (60 -> 61) (discovery-trial183). 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO state.py:1684 (2): Start update from 60 to 61 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8b96a726-7bf9-4da5-9239-52b473dec480 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:53.524950] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO grpc.py:2535 (2): Received request to add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage42, context: None 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:53.529377] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage42 2026-02-24T16:43:53.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:53.529400] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:53.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] INFO state.py:2068 (2): Update complete (60 -> 61) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO state.py:1684 (2): Start update from 60 to 61 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8b96a726-7bf9-4da5-9239-52b473dec480 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO state.py:1684 (126): Start update from 60 to 61 (discovery-trial191). 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO state.py:2068 (126): Update complete (60 -> 61) (discovery-trial191). 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:53.527665] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.745 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO grpc.py:2535 (2): Received request to add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage42, context: None 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:53.532098] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage42 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:53.532119] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] INFO state.py:2068 (2): Update complete (60 -> 61) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.746 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-mon[44017]: pgmap v125: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.7 KiB/s wr, 1.38k op/s 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO state.py:1684 (2): Start update from 60 to 61 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO state.py:1684 (132): Start update from 60 to 61 (discovery-trial153). 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO state.py:2068 (132): Update complete (60 -> 61) (discovery-trial153). 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO grpc.py:2774 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8b96a726-7bf9-4da5-9239-52b473dec480 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:53.518477] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO grpc.py:2535 (2): Received request to add bdev_8b96a726-7bf9-4da5-9239-52b473dec480 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage42, context: None 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:53.522966] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage42 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:53.522989] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 2 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] INFO state.py:2068 (2): Update complete (60 -> 61) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:53.746 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.831 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:53.850 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:53.850 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:53.850 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage44 2026-02-24T16:43:53.850 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage44 --load-balancing-group 1 2026-02-24T16:43:53.882 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:53Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.693634] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO grpc.py:2535 (2): Received request to add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage43, context: , client address: IPv4 10.20.193.51:45050 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.719767] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage43 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.719790] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_3, OMAP version: 62 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:53] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.832032] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:53.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:53 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:53.832065] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:54.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:53 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:53] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:53 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:53] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:53 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:53] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:1684 (81): Start update from 61 to 62 (discovery-trial151). 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:2068 (81): Update complete (61 -> 62) (discovery-trial151). 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:38420 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage44, no need to remove 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:38420 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:54.423636] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage44, context: , client address: IPv4 10.20.193.51:38420 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:54.431799] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage44 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:54.431820] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_4, OMAP version: 63 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:54.516088] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:54.513 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:54.516113] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:54.515 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:54.536 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:54.536 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:54.536 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage45 2026-02-24T16:43:54.536 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage45 --load-balancing-group 2 2026-02-24T16:43:54.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:1684 (132): Start update from 61 to 62 (discovery-trial153). 2026-02-24T16:43:54.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:2068 (132): Update complete (61 -> 62) (discovery-trial153). 2026-02-24T16:43:54.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 61 to 62 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.279090] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage43, context: None 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.283454] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage43 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.283476] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:2068 (2): Update complete (61 -> 62) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.575 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:54Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 61 to 62 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:1684 (126): Start update from 61 to 62 (discovery-trial191). 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:2068 (126): Update complete (61 -> 62) (discovery-trial191). 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.284667] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage43, context: None 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.289049] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage43 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.289071] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:2068 (2): Update complete (61 -> 62) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:1684 (105): Start update from 61 to 62 (discovery-trial183). 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:2068 (105): Update complete (61 -> 62) (discovery-trial183). 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 61 to 62 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.281942] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_28c3b9d1-254e-42d7-86d2-bd3ecc9709f3 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage43, context: None 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.286351] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage43 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.286373] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 3 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:2068 (2): Update complete (61 -> 62) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:54.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:54.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:1684 (81): Start update from 62 to 63 (discovery-trial151). 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:54 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:54] INFO state.py:2068 (81): Update complete (62 -> 63) (discovery-trial151). 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:24024 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage45, no need to remove 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:24024 2026-02-24T16:43:55.221 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.132636] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:2535 (2): Received request to add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage45, context: , client address: IPv4 10.20.193.51:24024 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.137129] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage45 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.137153] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_5, OMAP version: 64 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.224219] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:55.222 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.224251] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:55.224 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 62 to 63 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:1684 (105): Start update from 62 to 63 (discovery-trial183). 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO state.py:2068 (105): Update complete (62 -> 63) (discovery-trial183). 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.970085] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage44, context: None 2026-02-24T16:43:55.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.974460] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage44 2026-02-24T16:43:55.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:54 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:54.974483] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:55.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (62 -> 63) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:55.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.245 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:55.245 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:55.245 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage46 2026-02-24T16:43:55.246 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage46 --load-balancing-group 3 2026-02-24T16:43:55.285 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:55Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:55.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:1684 (132): Start update from 62 to 63 (discovery-trial153). 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:2068 (132): Update complete (62 -> 63) (discovery-trial153). 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 62 to 63 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.972370] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage44, context: None 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.976751] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage44 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:54 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:54.976773] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:55.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (62 -> 63) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:55.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:1684 (2): Start update from 62 to 63 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:2774 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:1684 (126): Start update from 62 to 63 (discovery-trial191). 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO state.py:2068 (126): Update complete (62 -> 63) (discovery-trial191). 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.977671] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:54] INFO grpc.py:2535 (2): Received request to add bdev_3a5dd3f8-cf76-4fe9-afb4-e6b457fa19d6 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage44, context: None 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.982034] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage44 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:54.982057] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 4 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (62 -> 63) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.484 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-mon[39453]: pgmap v126: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 960 KiB/s rd, 4.7 KiB/s wr, 956 op/s 2026-02-24T16:43:55.484 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/641152158' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:55.644 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-mon[44017]: pgmap v126: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 960 KiB/s rd, 4.7 KiB/s wr, 956 op/s 2026-02-24T16:43:55.645 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/641152158' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:55.647 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-mon[43553]: pgmap v126: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 960 KiB/s rd, 4.7 KiB/s wr, 956 op/s 2026-02-24T16:43:55.647 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/641152158' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:1684 (81): Start update from 63 to 64 (discovery-trial151). 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:2068 (81): Update complete (63 -> 64) (discovery-trial151). 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:45466 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage46, no need to remove 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:45466 2026-02-24T16:43:55.902 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.817361] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO grpc.py:2535 (2): Received request to add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage46, context: , client address: IPv4 10.20.193.51:45466 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.821785] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage46 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.821806] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_6, OMAP version: 65 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:55] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.905470] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:55.903 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:55 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:55.905496] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:55.905 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:55.926 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:55.926 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:55.926 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage47 2026-02-24T16:43:55.926 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage47 --load-balancing-group 4 2026-02-24T16:43:55.962 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:55Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO state.py:1684 (105): Start update from 63 to 64 (discovery-trial183). 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO state.py:2068 (105): Update complete (63 -> 64) (discovery-trial183). 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO state.py:1684 (2): Start update from 63 to 64 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:55.680268] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO grpc.py:2535 (2): Received request to add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage45, context: None 2026-02-24T16:43:55.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:55.684800] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage45 2026-02-24T16:43:55.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:55.684824] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:55.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (63 -> 64) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:55.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:55.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:55 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:55] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO state.py:1684 (132): Start update from 63 to 64 (discovery-trial153). 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO state.py:2068 (132): Update complete (63 -> 64) (discovery-trial153). 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO state.py:1684 (2): Start update from 63 to 64 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:55.676530] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO grpc.py:2535 (2): Received request to add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage45, context: None 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:55.681068] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage45 2026-02-24T16:43:56.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:55.681090] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:56.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (63 -> 64) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:56.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:55 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO state.py:1684 (2): Start update from 63 to 64 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO grpc.py:2774 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO state.py:1684 (126): Start update from 63 to 64 (discovery-trial191). 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO state.py:2068 (126): Update complete (63 -> 64) (discovery-trial191). 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:55.686322] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO grpc.py:2535 (2): Received request to add bdev_7e2595e1-3547-4c9f-8c23-62f4df57a5bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 5, auto_visible: True, auto_resize: True, RBD image mypool/myimage45, context: None 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:55.690699] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage45 2026-02-24T16:43:56.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:55.690722] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 5 2026-02-24T16:43:56.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] INFO state.py:2068 (2): Update complete (63 -> 64) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:56.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:55 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:55] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.578 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:56.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO state.py:1684 (2): Start update from 64 to 65 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:56.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:56.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:56.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO state.py:1684 (126): Start update from 64 to 65 (discovery-trial191). 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO state.py:2068 (126): Update complete (64 -> 65) (discovery-trial191). 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:56.366935] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO grpc.py:2535 (2): Received request to add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage46, context: None 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:56.371330] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage46 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:56.371353] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] INFO state.py:2068 (2): Update complete (64 -> 65) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:56 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:56] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.599 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:56.599 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:56.599 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage48 2026-02-24T16:43:56.599 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage48 --load-balancing-group 1 2026-02-24T16:43:56.631 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:56Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO state.py:1684 (2): Start update from 64 to 65 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO state.py:1684 (105): Start update from 64 to 65 (discovery-trial183). 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO state.py:2068 (105): Update complete (64 -> 65) (discovery-trial183). 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:56.361249] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO grpc.py:2535 (2): Received request to add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage46, context: None 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:56.369426] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage46 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:56.369447] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] INFO state.py:2068 (2): Update complete (64 -> 65) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:56 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:56] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO state.py:1684 (81): Start update from 64 to 65 (discovery-trial151). 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO state.py:2068 (81): Update complete (64 -> 65) (discovery-trial151). 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:40892 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage47, no need to remove 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:40892 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:56.493521] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO grpc.py:2535 (2): Received request to add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage47, context: , client address: IPv4 10.20.193.51:40892 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:56.497951] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage47 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:56.497973] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_7, OMAP version: 66 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:56] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:56.578496] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:56.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:56 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:56.578522] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO state.py:1684 (132): Start update from 64 to 65 (discovery-trial153). 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO state.py:2068 (132): Update complete (64 -> 65) (discovery-trial153). 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO state.py:1684 (2): Start update from 64 to 65 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO grpc.py:2774 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:56.360933] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO grpc.py:2535 (2): Received request to add bdev_8309032c-1aea-458a-8fd6-d156a3fd0e03 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 6, auto_visible: True, auto_resize: True, RBD image mypool/myimage46, context: None 2026-02-24T16:43:56.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:56.365332] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage46 2026-02-24T16:43:56.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:56.365354] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 6 2026-02-24T16:43:56.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] INFO state.py:2068 (2): Update complete (64 -> 65) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:56.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:56.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:56 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:56] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.251 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:57.272 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:57.272 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:57.272 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage49 2026-02-24T16:43:57.272 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage49 --load-balancing-group 2 2026-02-24T16:43:57.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:1684 (132): Start update from 65 to 66 (discovery-trial153). 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:2068 (132): Update complete (65 -> 66) (discovery-trial153). 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 65 to 66 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.034437] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage47, context: None 2026-02-24T16:43:57.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.038911] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage47 2026-02-24T16:43:57.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.038936] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:57.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (65 -> 66) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:57.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.318 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:57Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 65 to 66 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:1684 (126): Start update from 65 to 66 (discovery-trial191). 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:2068 (126): Update complete (65 -> 66) (discovery-trial191). 2026-02-24T16:43:57.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.043030] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage47, context: None 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.047511] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage47 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.047535] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (65 -> 66) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:1684 (81): Start update from 65 to 66 (discovery-trial151). 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:2068 (81): Update complete (65 -> 66) (discovery-trial151). 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:33886 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage48, no need to remove 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_04c36010-6526-485a-9329-6916b03ccedb from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:33886 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.171047] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_04c36010-6526-485a-9329-6916b03ccedb rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_04c36010-6526-485a-9329-6916b03ccedb to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage48, context: , client address: IPv4 10.20.193.51:33886 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.175394] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage48 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.175417] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_8, OMAP version: 67 2026-02-24T16:43:57.377 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.251356] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.251385] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:1684 (105): Start update from 65 to 66 (discovery-trial183). 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:2068 (105): Update complete (65 -> 66) (discovery-trial183). 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 65 to 66 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.037251] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_c98b8285-98f9-4f4f-9dd5-5a800f1276e1 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 7, auto_visible: True, auto_resize: True, RBD image mypool/myimage47, context: None 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.047429] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage47 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.047450] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 7 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (65 -> 66) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.378 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.685 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/426694902' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.685 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/68011545' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.685 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-mon[43553]: pgmap v127: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.27k op/s 2026-02-24T16:43:57.685 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/426694902' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.686 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/68011545' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.686 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-mon[44017]: pgmap v127: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.27k op/s 2026-02-24T16:43:57.687 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/426694902' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.687 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/68011545' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:57.687 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-mon[39453]: pgmap v127: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.27k op/s 2026-02-24T16:43:57.929 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:57.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 66 to 67 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_04c36010-6526-485a-9329-6916b03ccedb from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:1684 (105): Start update from 66 to 67 (discovery-trial183). 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:2068 (105): Update complete (66 -> 67) (discovery-trial183). 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.714635] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_04c36010-6526-485a-9329-6916b03ccedb rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_04c36010-6526-485a-9329-6916b03ccedb to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage48, context: None 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.719022] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage48 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:57.719045] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:57.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (66 -> 67) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:57.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:57 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:1684 (81): Start update from 66 to 67 (discovery-trial151). 2026-02-24T16:43:57.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:2068 (81): Update complete (66 -> 67) (discovery-trial151). 2026-02-24T16:43:57.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:43632 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage49, no need to remove 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:43632 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.839690] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage49, context: , client address: IPv4 10.20.193.51:43632 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.844102] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage49 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.844122] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_9, OMAP version: 68 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:57.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.929675] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:57.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:57 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:57.929704] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:58.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:1684 (132): Start update from 66 to 67 (discovery-trial153). 2026-02-24T16:43:58.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:2068 (132): Update complete (66 -> 67) (discovery-trial153). 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 66 to 67 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_04c36010-6526-485a-9329-6916b03ccedb from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.716484] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_04c36010-6526-485a-9329-6916b03ccedb rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_04c36010-6526-485a-9329-6916b03ccedb to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage48, context: None 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.720888] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage48 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:57.720909] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (66 -> 67) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:57 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:1684 (2): Start update from 66 to 67 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:2774 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_04c36010-6526-485a-9329-6916b03ccedb from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:1684 (126): Start update from 66 to 67 (discovery-trial191). 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:2068 (126): Update complete (66 -> 67) (discovery-trial191). 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.721066] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_04c36010-6526-485a-9329-6916b03ccedb rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO grpc.py:2535 (2): Received request to add bdev_04c36010-6526-485a-9329-6916b03ccedb to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 8, auto_visible: True, auto_resize: True, RBD image mypool/myimage48, context: None 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.725456] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage48 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:57.725479] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 8 2026-02-24T16:43:58.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] INFO state.py:2068 (2): Update complete (66 -> 67) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:58.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:57 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:57] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.182 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:58.182 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:58.182 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage50 2026-02-24T16:43:58.182 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage50 --load-balancing-group 3 2026-02-24T16:43:58.279 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:58Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:58.618 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1841352818' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO state.py:1684 (81): Start update from 67 to 68 (discovery-trial151). 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO state.py:2068 (81): Update complete (67 -> 68) (discovery-trial151). 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:46454 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage50, no need to remove 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:58.619 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:46454 2026-02-24T16:43:58.654 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1841352818' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:58.654 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1841352818' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO state.py:1684 (132): Start update from 67 to 68 (discovery-trial153). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO state.py:2068 (132): Update complete (67 -> 68) (discovery-trial153). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO state.py:1684 (2): Start update from 67 to 68 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:58.382024] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO grpc.py:2535 (2): Received request to add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage49, context: None 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:58.386602] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage49 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:58.386624] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] INFO state.py:2068 (2): Update complete (67 -> 68) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO state.py:1684 (105): Start update from 67 to 68 (discovery-trial183). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO state.py:2068 (105): Update complete (67 -> 68) (discovery-trial183). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO state.py:1684 (2): Start update from 67 to 68 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:58.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:58.383738] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO grpc.py:2535 (2): Received request to add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage49, context: None 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:58.388119] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage49 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:58.388142] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] INFO state.py:2068 (2): Update complete (67 -> 68) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO state.py:1684 (2): Start update from 67 to 68 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO grpc.py:2774 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO state.py:1684 (126): Start update from 67 to 68 (discovery-trial191). 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO state.py:2068 (126): Update complete (67 -> 68) (discovery-trial191). 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:58.390681] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO grpc.py:2535 (2): Received request to add bdev_e44b0407-8e5a-4024-9fd0-d27c03edf6a8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 9, auto_visible: True, auto_resize: True, RBD image mypool/myimage49, context: None 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:58.395110] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage49 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:58.395132] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 9 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] INFO state.py:2068 (2): Update complete (67 -> 68) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:58.656 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.732 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:58.753 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:58.753 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:58.753 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage51 2026-02-24T16:43:58.753 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage51 --load-balancing-group 4 2026-02-24T16:43:58.785 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:58Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:58.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:58 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:58] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:58.646396] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO grpc.py:2535 (2): Received request to add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage50, context: , client address: IPv4 10.20.193.51:46454 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:58.650832] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage50 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:58.650853] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_10, OMAP version: 69 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:58] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:58.732077] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:58.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:58 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:58.732104] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:59.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:58 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:58] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:58 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:58] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:1684 (132): Start update from 68 to 69 (discovery-trial153). 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:2068 (132): Update complete (68 -> 69) (discovery-trial153). 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 68 to 69 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.190848] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage50, context: None 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.195493] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage50 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.195514] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:59.383 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (68 -> 69) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:43:59.467 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-mon[39453]: pgmap v128: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 6.5 KiB/s wr, 1.16k op/s 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:1684 (81): Start update from 68 to 69 (discovery-trial151). 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:2068 (81): Update complete (68 -> 69) (discovery-trial151). 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:53994 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage51, no need to remove 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:53994 2026-02-24T16:43:59.467 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:59.340633] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage51, context: , client address: IPv4 10.20.193.51:53994 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:59.364780] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage51 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:59.364801] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_11, OMAP version: 70 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:59.469845] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:43:59.468 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:43:59.469869] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:43:59.469 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:43:59.483 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-mon[43553]: pgmap v128: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 6.5 KiB/s wr, 1.16k op/s 2026-02-24T16:43:59.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:1684 (105): Start update from 68 to 69 (discovery-trial183). 2026-02-24T16:43:59.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:2068 (105): Update complete (68 -> 69) (discovery-trial183). 2026-02-24T16:43:59.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 68 to 69 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.188078] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage50, context: None 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.192442] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage50 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.192462] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (68 -> 69) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.491 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:43:59.491 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:43:59.491 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage52 2026-02-24T16:43:59.491 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage52 --load-balancing-group 1 2026-02-24T16:43:59.525 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:43:59Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 68 to 69 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:1684 (126): Start update from 68 to 69 (discovery-trial191). 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:2068 (126): Update complete (68 -> 69) (discovery-trial191). 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.199568] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_142f883b-d314-4f53-aaf8-d56e636cc5e8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 10, auto_visible: True, auto_resize: True, RBD image mypool/myimage50, context: None 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.204029] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage50 2026-02-24T16:43:59.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.204053] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 10 2026-02-24T16:43:59.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (68 -> 69) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:43:59.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-mon[44017]: pgmap v128: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 6.5 KiB/s wr, 1.16k op/s 2026-02-24T16:43:59.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:43:59.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:1684 (81): Start update from 69 to 70 (discovery-trial151). 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO state.py:2068 (81): Update complete (69 -> 70) (discovery-trial151). 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:43:59 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:38090 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage52, no need to remove 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b0854e12-150b-430e-98fb-f47be6a0b321 from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:38090 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.082651] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO grpc.py:2535 (2): Received request to add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage52, context: , client address: IPv4 10.20.193.51:38090 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.087204] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage52 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.087226] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_12, OMAP version: 71 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.176320] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:00.173 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.176348] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:00.175 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:00.197 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:00.197 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:00.198 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage53 2026-02-24T16:44:00.198 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage53 --load-balancing-group 2 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 69 to 70 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:1684 (105): Start update from 69 to 70 (discovery-trial183). 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:2068 (105): Update complete (69 -> 70) (discovery-trial183). 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.924112] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage51, context: None 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.928455] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage51 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:43:59.928477] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:43:59 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (69 -> 70) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.237 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:00Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:1684 (132): Start update from 69 to 70 (discovery-trial153). 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:2068 (132): Update complete (69 -> 70) (discovery-trial153). 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 69 to 70 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.922709] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage51, context: None 2026-02-24T16:44:00.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.927660] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage51 2026-02-24T16:44:00.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:43:59.927679] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:44:00.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:43:59 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (69 -> 70) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:00.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:1684 (2): Start update from 69 to 70 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:2774 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:1684 (126): Start update from 69 to 70 (discovery-trial191). 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:2068 (126): Update complete (69 -> 70) (discovery-trial191). 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.931712] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO grpc.py:2535 (2): Received request to add bdev_1c7e3336-092f-4916-bba8-0f00dd7b20bd to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 11, auto_visible: True, auto_resize: True, RBD image mypool/myimage51, context: None 2026-02-24T16:44:00.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.936095] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage51 2026-02-24T16:44:00.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:43:59.936122] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 11 2026-02-24T16:44:00.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:43:59 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:43:59] INFO state.py:2068 (2): Update complete (69 -> 70) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:00.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:1684 (81): Start update from 70 to 71 (discovery-trial151). 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:2068 (81): Update complete (70 -> 71) (discovery-trial151). 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:60648 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage53, no need to remove 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:60648 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.796656] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO grpc.py:2535 (2): Received request to add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage53, context: , client address: IPv4 10.20.193.51:60648 2026-02-24T16:44:00.890 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.801063] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage53 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.801085] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_13, OMAP version: 72 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.893492] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:00.891 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:00.893519] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:00.893 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:00.915 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:00.916 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:00.916 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage54 2026-02-24T16:44:00.916 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage54 --load-balancing-group 3 2026-02-24T16:44:00.954 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:00Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO state.py:1684 (105): Start update from 70 to 71 (discovery-trial183). 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO state.py:2068 (105): Update complete (70 -> 71) (discovery-trial183). 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO state.py:1684 (2): Start update from 70 to 71 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b0854e12-150b-430e-98fb-f47be6a0b321 from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:00.625698] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO grpc.py:2535 (2): Received request to add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage52, context: None 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:00.630101] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage52 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:00.630120] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] INFO state.py:2068 (2): Update complete (70 -> 71) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:00.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:00 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO state.py:1684 (2): Start update from 70 to 71 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b0854e12-150b-430e-98fb-f47be6a0b321 from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO state.py:1684 (132): Start update from 70 to 71 (discovery-trial153). 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO state.py:2068 (132): Update complete (70 -> 71) (discovery-trial153). 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:00.627263] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO grpc.py:2535 (2): Received request to add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage52, context: None 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:00.631636] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage52 2026-02-24T16:44:01.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:00.631658] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:44:01.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] INFO state.py:2068 (2): Update complete (70 -> 71) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:01.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:00 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO state.py:1684 (2): Start update from 70 to 71 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO grpc.py:2774 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b0854e12-150b-430e-98fb-f47be6a0b321 from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO state.py:1684 (126): Start update from 70 to 71 (discovery-trial191). 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO state.py:2068 (126): Update complete (70 -> 71) (discovery-trial191). 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:00.632436] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO grpc.py:2535 (2): Received request to add bdev_b0854e12-150b-430e-98fb-f47be6a0b321 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 12, auto_visible: True, auto_resize: True, RBD image mypool/myimage52, context: None 2026-02-24T16:44:01.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:00.636794] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage52 2026-02-24T16:44:01.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:00.636818] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 12 2026-02-24T16:44:01.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] INFO state.py:2068 (2): Update complete (70 -> 71) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:01.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:00 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:00] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.563 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-mon[44017]: pgmap v129: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 4.8 KiB/s wr, 1.02k op/s 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO state.py:1684 (132): Start update from 71 to 72 (discovery-trial153). 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO state.py:2068 (132): Update complete (71 -> 72) (discovery-trial153). 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO state.py:1684 (2): Start update from 71 to 72 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:01.340477] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO grpc.py:2535 (2): Received request to add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage53, context: None 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:01.344880] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage53 2026-02-24T16:44:01.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:01.344900] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:44:01.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] INFO state.py:2068 (2): Update complete (71 -> 72) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:01.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:01 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:01] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.576 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-mon[39453]: pgmap v129: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 4.8 KiB/s wr, 1.02k op/s 2026-02-24T16:44:01.576 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO state.py:1684 (81): Start update from 71 to 72 (discovery-trial151). 2026-02-24T16:44:01.576 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO state.py:2068 (81): Update complete (71 -> 72) (discovery-trial151). 2026-02-24T16:44:01.576 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:60892 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage54, no need to remove 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:60892 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:01.489405] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO grpc.py:2535 (2): Received request to add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage54, context: , client address: IPv4 10.20.193.51:60892 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:01.493797] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage54 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:01.493819] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_14, OMAP version: 73 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:01] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:01.579120] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:01.577 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:01 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:01.579146] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:01.580 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:01.590 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO state.py:1684 (2): Start update from 71 to 72 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO state.py:1684 (126): Start update from 71 to 72 (discovery-trial191). 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO state.py:2068 (126): Update complete (71 -> 72) (discovery-trial191). 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:01.345306] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO grpc.py:2535 (2): Received request to add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage53, context: None 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:01.349639] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage53 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:01.349668] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] INFO state.py:2068 (2): Update complete (71 -> 72) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:01.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:01 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:01] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.601 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:01.601 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:01.601 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage55 2026-02-24T16:44:01.601 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage55 --load-balancing-group 4 2026-02-24T16:44:01.640 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:01Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:01.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-mon[43553]: pgmap v129: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 4.8 KiB/s wr, 1.02k op/s 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO state.py:1684 (105): Start update from 71 to 72 (discovery-trial183). 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO state.py:2068 (105): Update complete (71 -> 72) (discovery-trial183). 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO state.py:1684 (2): Start update from 71 to 72 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO grpc.py:2774 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:01.338026] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO grpc.py:2535 (2): Received request to add bdev_608a7ae4-773e-4ef4-907e-7149bcc633ca to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 13, auto_visible: True, auto_resize: True, RBD image mypool/myimage53, context: None 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:01.342526] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage53 2026-02-24T16:44:01.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:01.342549] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 13 2026-02-24T16:44:01.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] INFO state.py:2068 (2): Update complete (71 -> 72) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:01.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:01.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:01 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:01] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.283 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:1684 (81): Start update from 72 to 73 (discovery-trial151). 2026-02-24T16:44:02.283 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:2068 (81): Update complete (72 -> 73) (discovery-trial151). 2026-02-24T16:44:02.283 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:46450 2026-02-24T16:44:02.283 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage55, no need to remove 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:46450 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.205403] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage55, context: , client address: IPv4 10.20.193.51:46450 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.209779] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage55 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.209800] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_15, OMAP version: 74 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.284 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.285993] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:02.285 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.286014] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:02.285 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:1684 (132): Start update from 72 to 73 (discovery-trial153). 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:2068 (132): Update complete (72 -> 73) (discovery-trial153). 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 72 to 73 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.030221] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage54, context: None 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.034981] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage54 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.035006] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (72 -> 73) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 72 to 73 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:1684 (126): Start update from 72 to 73 (discovery-trial191). 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:2068 (126): Update complete (72 -> 73) (discovery-trial191). 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.042333] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage54, context: None 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.048519] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage54 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.048543] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (72 -> 73) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:02.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.387 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 72 to 73 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:02.387 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:1684 (105): Start update from 72 to 73 (discovery-trial183). 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:2068 (105): Update complete (72 -> 73) (discovery-trial183). 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.034521] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_7b358c27-f143-44e8-a648-5edbc32d6d46 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 14, auto_visible: True, auto_resize: True, RBD image mypool/myimage54, context: None 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.038905] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage54 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.038924] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 14 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (72 -> 73) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.388 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.516 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:02.516 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:02.516 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage56 2026-02-24T16:44:02.516 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage56 --load-balancing-group 1 2026-02-24T16:44:02.598 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:02Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:02.719 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2363089389' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:02.720 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2363089389' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:02.722 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2363089389' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:02.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:1684 (81): Start update from 73 to 74 (discovery-trial151). 2026-02-24T16:44:02.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:2068 (81): Update complete (73 -> 74) (discovery-trial151). 2026-02-24T16:44:02.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:33182 2026-02-24T16:44:02.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage56, no need to remove 2026-02-24T16:44:02.980 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:02.980 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:02.980 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:33182 2026-02-24T16:44:02.980 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.982024] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 73 to 74 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:02.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:1684 (105): Start update from 73 to 74 (discovery-trial183). 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:2068 (105): Update complete (73 -> 74) (discovery-trial183). 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.743303] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage55, context: None 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.747717] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage55 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:02.747739] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:44:02.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (73 -> 74) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:02.990 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 73 to 74 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:02.990 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.990 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.990 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:1684 (126): Start update from 73 to 74 (discovery-trial191). 2026-02-24T16:44:02.990 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:2068 (126): Update complete (73 -> 74) (discovery-trial191). 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.754329] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage55, context: None 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.758831] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage55 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:02.758855] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:02 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (73 -> 74) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:02.991 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:1684 (132): Start update from 73 to 74 (discovery-trial153). 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:2068 (132): Update complete (73 -> 74) (discovery-trial153). 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:1684 (2): Start update from 73 to 74 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:2774 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.750813] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_4e5113f1-ae3e-4036-9ede-fd1641d29daa to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 15, auto_visible: True, auto_resize: True, RBD image mypool/myimage55, context: None 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.755169] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage55 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:02.755192] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 15 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] INFO state.py:2068 (2): Update complete (73 -> 74) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:02.992 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.070 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:03.094 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:03.094 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:03.095 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage57 2026-02-24T16:44:03.095 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage57 --load-balancing-group 2 2026-02-24T16:44:03.134 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:03Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO grpc.py:2535 (2): Received request to add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage56, context: , client address: IPv4 10.20.193.51:33182 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.986436] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage56 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:02.986458] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_16, OMAP version: 75 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:02 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:02] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.069764] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:03.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.069788] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:03.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:02 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.392 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.392 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:02 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:02] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.672 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-mon[39453]: pgmap v130: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.5 KiB/s wr, 1.23k op/s 2026-02-24T16:44:03.672 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO state.py:1684 (81): Start update from 74 to 75 (discovery-trial151). 2026-02-24T16:44:03.672 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO state.py:2068 (81): Update complete (74 -> 75) (discovery-trial151). 2026-02-24T16:44:03.673 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:55656 2026-02-24T16:44:03.673 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage57, no need to remove 2026-02-24T16:44:03.673 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:03.673 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:03.673 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3c219351-ecdb-43be-a263-14ffde10208a from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:55656 2026-02-24T16:44:03.702 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-mon[44017]: pgmap v130: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.5 KiB/s wr, 1.23k op/s 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO state.py:1684 (132): Start update from 74 to 75 (discovery-trial153). 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO state.py:2068 (132): Update complete (74 -> 75) (discovery-trial153). 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO state.py:1684 (2): Start update from 74 to 75 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:03.523698] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO grpc.py:2535 (2): Received request to add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage56, context: None 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:03.528107] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage56 2026-02-24T16:44:03.703 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:03.528134] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:44:03.704 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] INFO state.py:2068 (2): Update complete (74 -> 75) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:03.704 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.704 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-mon[43553]: pgmap v130: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 6.5 KiB/s wr, 1.23k op/s 2026-02-24T16:44:03.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO state.py:1684 (2): Start update from 74 to 75 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO state.py:1684 (105): Start update from 74 to 75 (discovery-trial183). 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO state.py:2068 (105): Update complete (74 -> 75) (discovery-trial183). 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:03.527085] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO grpc.py:2535 (2): Received request to add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage56, context: None 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:03.531600] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage56 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:03.531621] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] INFO state.py:2068 (2): Update complete (74 -> 75) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:03.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.790 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:03.812 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:03.812 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:03.812 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage58 2026-02-24T16:44:03.812 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage58 --load-balancing-group 3 2026-02-24T16:44:03.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO state.py:1684 (2): Start update from 74 to 75 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO grpc.py:2774 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO state.py:1684 (126): Start update from 74 to 75 (discovery-trial191). 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO state.py:2068 (126): Update complete (74 -> 75) (discovery-trial191). 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:03.531311] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO grpc.py:2535 (2): Received request to add bdev_4c615bb7-4edb-4cc1-92b5-3c599225e672 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 16, auto_visible: True, auto_resize: True, RBD image mypool/myimage56, context: None 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:03.535689] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage56 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:03.535713] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 16 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] INFO state.py:2068 (2): Update complete (74 -> 75) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:03 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:03] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.852 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:03Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:03.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:03 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:03] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.694240] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3c219351-ecdb-43be-a263-14ffde10208a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO grpc.py:2535 (2): Received request to add bdev_3c219351-ecdb-43be-a263-14ffde10208a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/myimage57, context: , client address: IPv4 10.20.193.51:55656 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.698664] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage57 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.698685] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_17, OMAP version: 76 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:03] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.790756] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:03.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:03 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:03.790779] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:04.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:03 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:03] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:1684 (105): Start update from 75 to 76 (discovery-trial183). 2026-02-24T16:44:04.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:2068 (105): Update complete (75 -> 76) (discovery-trial183). 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 75 to 76 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3c219351-ecdb-43be-a263-14ffde10208a from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.239975] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3c219351-ecdb-43be-a263-14ffde10208a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_3c219351-ecdb-43be-a263-14ffde10208a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage57, context: None 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.244319] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage57 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.244339] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:2068 (2): Update complete (75 -> 76) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:1684 (81): Start update from 75 to 76 (discovery-trial151). 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:2068 (81): Update complete (75 -> 76) (discovery-trial151). 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:11880 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage58, no need to remove 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_99635a96-175a-4876-9a62-b370d6a4fe90 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:11880 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:04.423187] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/myimage58, context: , client address: IPv4 10.20.193.51:11880 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:04.427588] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage58 2026-02-24T16:44:04.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:04.427614] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:44:04.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_18, OMAP version: 77 2026-02-24T16:44:04.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.521 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:04.543 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:04.543 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:04.543 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage59 2026-02-24T16:44:04.543 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage59 --load-balancing-group 4 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:1684 (132): Start update from 75 to 76 (discovery-trial153). 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:2068 (132): Update complete (75 -> 76) (discovery-trial153). 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 75 to 76 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3c219351-ecdb-43be-a263-14ffde10208a from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.236279] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3c219351-ecdb-43be-a263-14ffde10208a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_3c219351-ecdb-43be-a263-14ffde10208a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage57, context: None 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.240645] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage57 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.240676] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:2068 (2): Update complete (75 -> 76) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.581 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:04Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 75 to 76 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_3c219351-ecdb-43be-a263-14ffde10208a from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:1684 (126): Start update from 75 to 76 (discovery-trial191). 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:2068 (126): Update complete (75 -> 76) (discovery-trial191). 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.243764] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_3c219351-ecdb-43be-a263-14ffde10208a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_3c219351-ecdb-43be-a263-14ffde10208a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 17, auto_visible: True, auto_resize: True, RBD image mypool/myimage57, context: None 2026-02-24T16:44:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.248165] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage57 2026-02-24T16:44:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.248187] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 17 2026-02-24T16:44:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:2068 (2): Update complete (75 -> 76) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:04.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:04.521734] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:04.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:04.521759] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:05.212 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:1684 (81): Start update from 76 to 77 (discovery-trial151). 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:04 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:04] INFO state.py:2068 (81): Update complete (76 -> 77) (discovery-trial151). 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:53976 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage59, no need to remove 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:53976 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.118277] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:2535 (2): Received request to add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/myimage59, context: , client address: IPv4 10.20.193.51:53976 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.122699] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage59 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.122724] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_19, OMAP version: 78 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.215836] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:05.213 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.215861] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:05.216 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 76 to 77 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_99635a96-175a-4876-9a62-b370d6a4fe90 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:1684 (105): Start update from 76 to 77 (discovery-trial183). 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO state.py:2068 (105): Update complete (76 -> 77) (discovery-trial183). 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.968252] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage58, context: None 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.972673] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage58 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:04 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:04.972694] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (76 -> 77) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.237 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:05.237 INFO:tasks.workunit.client.0.trial051.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2026-02-24T16:44:05.237 INFO:tasks.workunit.client.0.trial051.stderr:+ image=myimage60 2026-02-24T16:44:05.237 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage60 --load-balancing-group 1 2026-02-24T16:44:05.276 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:05Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:1684 (132): Start update from 76 to 77 (discovery-trial153). 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:2068 (132): Update complete (76 -> 77) (discovery-trial153). 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 76 to 77 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_99635a96-175a-4876-9a62-b370d6a4fe90 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.969107] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage58, context: None 2026-02-24T16:44:05.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.973474] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage58 2026-02-24T16:44:05.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:04 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:04.973503] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:44:05.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (76 -> 77) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:05.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:1684 (126): Start update from 76 to 77 (discovery-trial191). 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:2068 (126): Update complete (76 -> 77) (discovery-trial191). 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO state.py:1684 (2): Start update from 76 to 77 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:2774 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_99635a96-175a-4876-9a62-b370d6a4fe90 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.975917] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:04] INFO grpc.py:2535 (2): Received request to add bdev_99635a96-175a-4876-9a62-b370d6a4fe90 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 18, auto_visible: True, auto_resize: True, RBD image mypool/myimage58, context: None 2026-02-24T16:44:05.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.980313] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage58 2026-02-24T16:44:05.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:04 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:04.980334] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 18 2026-02-24T16:44:05.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (76 -> 77) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:05.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.484 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-mon[39453]: pgmap v131: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 994 KiB/s rd, 4.8 KiB/s wr, 925 op/s 2026-02-24T16:44:05.656 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-mon[44017]: pgmap v131: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 994 KiB/s rd, 4.8 KiB/s wr, 925 op/s 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO state.py:1684 (132): Start update from 77 to 78 (discovery-trial153). 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO state.py:2068 (132): Update complete (77 -> 78) (discovery-trial153). 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO state.py:1684 (2): Start update from 77 to 78 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:05.656 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:05.661953] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.659 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-mon[43553]: pgmap v131: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 994 KiB/s rd, 4.8 KiB/s wr, 925 op/s 2026-02-24T16:44:05.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO state.py:1684 (2): Start update from 77 to 78 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:05.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO state.py:1684 (105): Start update from 77 to 78 (discovery-trial183). 2026-02-24T16:44:05.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:05.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO state.py:2068 (105): Update complete (77 -> 78) (discovery-trial183). 2026-02-24T16:44:05.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:05.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:05.665164] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:1684 (81): Start update from 77 to 78 (discovery-trial151). 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:2068 (81): Update complete (77 -> 78) (discovery-trial151). 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.51:46138 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/myimage60, no need to remove 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:44:05.922 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.51:46138 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.832707] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO grpc.py:2535 (2): Received request to add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/myimage60, context: , client address: IPv4 10.20.193.51:46138 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.837268] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage60 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.837289] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_20, OMAP version: 79 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:05] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.924610] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:44:05.923 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:44:05.924635] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:44:05.924 INFO:tasks.workunit.client.0.trial051.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:44:05.945 INFO:tasks.workunit.client.0.trial051.stderr:+ (( image_index++ )) 2026-02-24T16:44:05.945 INFO:tasks.workunit.client.0.trial051.stderr:+ list_subsystems 2026-02-24T16:44:05.945 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:44:05.945 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.151 2026-02-24T16:44:05.946 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json subsystem list 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO grpc.py:2535 (2): Received request to add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage59, context: None 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:05.669545] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage59 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:05.669566] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (77 -> 78) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:05 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:05] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:05.987 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:05Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO grpc.py:2535 (2): Received request to add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage59, context: None 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:05.666384] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage59 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:05.666413] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (77 -> 78) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:06.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:05 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:05] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:06.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO state.py:1684 (2): Start update from 77 to 78 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:06.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO grpc.py:2774 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:06.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:06.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO state.py:1684 (126): Start update from 77 to 78 (discovery-trial191). 2026-02-24T16:44:06.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO state.py:2068 (126): Update complete (77 -> 78) (discovery-trial191). 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:05.668066] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO grpc.py:2535 (2): Received request to add bdev_8674d792-6821-465c-bc6a-a7cb2de67dd8 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 19, auto_visible: True, auto_resize: True, RBD image mypool/myimage59, context: None 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:05.672430] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage59 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:05.672455] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 19 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] INFO state.py:2068 (2): Update complete (77 -> 78) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:06.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:05 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:05] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout:{ 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "error_message": "Success", 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "subsystems": [ 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph96885192365298", 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 1, 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 2040, 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.454 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph54708660634751", 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 1, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 2040, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.455 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph87132565256667", 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 1, 2026-02-24T16:44:06.456 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 2040, 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.457 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.458 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.458 INFO:tasks.workunit.client.0.trial051.stdout: } 2026-02-24T16:44:06.458 INFO:tasks.workunit.client.0.trial051.stdout: ], 2026-02-24T16:44:06.458 INFO:tasks.workunit.client.0.trial051.stdout: "status": 0 2026-02-24T16:44:06.458 INFO:tasks.workunit.client.0.trial051.stdout:} 2026-02-24T16:44:06.480 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:44:06.480 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.153 2026-02-24T16:44:06.481 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.153 --server-port 5500 --format json subsystem list 2026-02-24T16:44:06.521 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:06Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:06.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1251176317' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO state.py:1684 (2): Start update from 78 to 79 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO state.py:1684 (105): Start update from 78 to 79 (discovery-trial183). 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO state.py:2068 (105): Update complete (78 -> 79) (discovery-trial183). 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:06.377832] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO grpc.py:2535 (2): Received request to add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage60, context: None 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:06.382360] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage60 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:44:06.382381] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:44:06.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:06] INFO state.py:2068 (2): Update complete (78 -> 79) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:44:06.735 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:06 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1251176317' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:06.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:06 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:06] INFO state.py:1684 (81): Start update from 78 to 79 (discovery-trial151). 2026-02-24T16:44:06.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:06 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:06] INFO state.py:2068 (81): Update complete (78 -> 79) (discovery-trial151). 2026-02-24T16:44:06.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:06 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:06] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.51:45762 2026-02-24T16:44:06.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1251176317' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO state.py:1684 (2): Start update from 78 to 79 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO state.py:1684 (132): Start update from 78 to 79 (discovery-trial153). 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO state.py:2068 (132): Update complete (78 -> 79) (discovery-trial153). 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:06.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:06.374165] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:06.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO grpc.py:2535 (2): Received request to add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage60, context: None 2026-02-24T16:44:06.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:06.378579] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage60 2026-02-24T16:44:06.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:44:06.378604] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:44:06.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO state.py:2068 (2): Update complete (78 -> 79) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:44:06.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO state.py:1684 (126): Start update from 78 to 79 (discovery-trial191). 2026-02-24T16:44:06.840 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO state.py:2068 (126): Update complete (78 -> 79) (discovery-trial191). 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO state.py:1684 (2): Start update from 78 to 79 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO grpc.py:2774 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:06.380610] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO grpc.py:2535 (2): Received request to add bdev_6b2cbf76-d023-4a57-a654-b59345cf9a2a to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 20, auto_visible: True, auto_resize: True, RBD image mypool/myimage60, context: None 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:06.385038] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name myimage60 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:44:06.385067] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 20 2026-02-24T16:44:06.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:06 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:06] INFO state.py:2068 (2): Update complete (78 -> 79) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:44:06.979 INFO:tasks.workunit.client.0.trial051.stdout:{ 2026-02-24T16:44:06.979 INFO:tasks.workunit.client.0.trial051.stdout: "error_message": "Success", 2026-02-24T16:44:06.979 INFO:tasks.workunit.client.0.trial051.stdout: "subsystems": [ 2026-02-24T16:44:06.979 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.979 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph96885192365298", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 2041, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 4080, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph54708660634751", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 2041, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 4080, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph87132565256667", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 2041, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 4080, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: } 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: ], 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout: "status": 0 2026-02-24T16:44:06.980 INFO:tasks.workunit.client.0.trial051.stdout:} 2026-02-24T16:44:07.000 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:44:07.000 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.183 2026-02-24T16:44:07.000 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.183 --server-port 5500 --format json subsystem list 2026-02-24T16:44:07.030 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:07.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:44:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:44:06] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.51:34012 2026-02-24T16:44:07.487 INFO:tasks.workunit.client.0.trial051.stdout:{ 2026-02-24T16:44:07.487 INFO:tasks.workunit.client.0.trial051.stdout: "error_message": "Success", 2026-02-24T16:44:07.487 INFO:tasks.workunit.client.0.trial051.stdout: "subsystems": [ 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph96885192365298", 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 4081, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 6120, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:07.488 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph54708660634751", 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 4081, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 6120, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:07.489 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph87132565256667", 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 4081, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 6120, 2026-02-24T16:44:07.490 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: } 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: ], 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout: "status": 0 2026-02-24T16:44:07.491 INFO:tasks.workunit.client.0.trial051.stdout:} 2026-02-24T16:44:07.507 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in "${!gateway_ips[@]}" 2026-02-24T16:44:07.507 INFO:tasks.workunit.client.0.trial051.stderr:+ ip=10.20.193.191 2026-02-24T16:44:07.507 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.191 --server-port 5500 --format json subsystem list 2026-02-24T16:44:07.552 INFO:tasks.workunit.client.0.trial051.stderr:time="2026-02-24T16:44:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:44:07.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:44:07 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:44:07] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.51:46152 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:07 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3663166510' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:07 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3400505197' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:07 trial183 ceph-mon[43553]: pgmap v132: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.22k op/s 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:07 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3663166510' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:07 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3400505197' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:07 trial151 ceph-mon[39453]: pgmap v132: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.22k op/s 2026-02-24T16:44:07.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:07 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3663166510' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:07 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3400505197' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:07.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:07 trial153 ceph-mon[44017]: pgmap v132: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.7 KiB/s wr, 1.22k op/s 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout:{ 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "error_message": "Success", 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "subsystems": [ 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph96885192365298", 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 6121, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 8160, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:08.021 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph54708660634751", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 6121, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 8160, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: }, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: { 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "enable_ha": true, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "serial_number": "Ceph87132565256667", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "model_number": "Ceph bdev Controller", 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "min_cntlid": 6121, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "max_cntlid": 8160, 2026-02-24T16:44:08.022 INFO:tasks.workunit.client.0.trial051.stdout: "namespace_count": 20, 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "subtype": "NVMe", 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "max_namespaces": 512, 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "has_dhchap_key": false, 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "allow_any_host": true, 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "created_without_key": false, 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "network_mask": [] 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: } 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: ], 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout: "status": 0 2026-02-24T16:44:08.023 INFO:tasks.workunit.client.0.trial051.stdout:} 2026-02-24T16:44:08.248 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Subsystem setup done 2026-02-24T16:44:08.248 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Subsystem setup done' 2026-02-24T16:44:08.248 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T16:44:08.248 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-24T16:44:08.318 INFO:tasks.workunit:Running workunits matching nvmeof/basic_tests.sh on client.0... 2026-02-24T16:44:08.319 INFO:tasks.workunit:Running workunit nvmeof/basic_tests.sh... 2026-02-24T16:44:08.319 DEBUG:teuthology.orchestra.run.trial051:workunit test nvmeof/basic_tests.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_IMAGE_PREFIX=myimage RBD_POOL=mypool timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/basic_tests.sh 2026-02-24T16:44:08.340 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:44:08 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:44:08] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.51:42708 2026-02-24T16:44:08.375 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo systemctl stop udisks2 2026-02-24T16:44:08.389 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo dnf install nvme-cli-2.13 libnvme-1.13 -y 2026-02-24T16:44:08.690 INFO:tasks.workunit.client.0.trial051.stdout:Last metadata expiration check: 0:06:17 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:44:08.774 INFO:tasks.workunit.client.0.trial051.stdout:Dependencies resolved. 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:Downgrading: 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout: libnvme x86_64 1.13-1.el9 baseos 111 k 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:Transaction Summary 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:Downgrade 2 Packages 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:Total download size: 1.2 M 2026-02-24T16:44:08.775 INFO:tasks.workunit.client.0.trial051.stdout:Downloading Packages: 2026-02-24T16:44:08.857 INFO:tasks.workunit.client.0.trial051.stdout:(1/2): libnvme-1.13-1.el9.x86_64.rpm 4.8 MB/s | 111 kB 00:00 2026-02-24T16:44:08.871 INFO:tasks.workunit.client.0.trial051.stdout:(2/2): nvme-cli-2.13-1.el9.x86_64.rpm 30 MB/s | 1.1 MB 00:00 2026-02-24T16:44:08.871 INFO:tasks.workunit.client.0.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:44:08.871 INFO:tasks.workunit.client.0.trial051.stdout:Total 13 MB/s | 1.2 MB 00:00 2026-02-24T16:44:08.890 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction check 2026-02-24T16:44:08.896 INFO:tasks.workunit.client.0.trial051.stdout:Transaction check succeeded. 2026-02-24T16:44:08.896 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction test 2026-02-24T16:44:08.940 INFO:tasks.workunit.client.0.trial051.stdout:Transaction test succeeded. 2026-02-24T16:44:08.940 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction 2026-02-24T16:44:09.006 INFO:tasks.workunit.client.0.trial051.stdout: Preparing : 1/1 2026-02-24T16:44:09.125 INFO:tasks.workunit.client.0.trial051.stdout: Downgrading : libnvme-1.13-1.el9.x86_64 1/4 2026-02-24T16:44:09.127 INFO:tasks.workunit.client.0.trial051.stdout: Downgrading : nvme-cli-2.13-1.el9.x86_64 2/4 2026-02-24T16:44:09.328 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 2/4 2026-02-24T16:44:09.332 INFO:tasks.workunit.client.0.trial051.stdout: Cleanup : nvme-cli-2.16-1.el9.x86_64 3/4 2026-02-24T16:44:09.357 INFO:tasks.workunit.client.0.trial051.stdout: Cleanup : libnvme-1.16.1-2.el9.x86_64 4/4 2026-02-24T16:44:09.638 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: libnvme-1.16.1-2.el9.x86_64 4/4 2026-02-24T16:44:09.639 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : libnvme-1.13-1.el9.x86_64 1/4 2026-02-24T16:44:09.639 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : libnvme-1.16.1-2.el9.x86_64 2/4 2026-02-24T16:44:09.639 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 3/4 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 4/4 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout:Downgraded: 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout: libnvme-1.13-1.el9.x86_64 nvme-cli-2.13-1.el9.x86_64 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:44:09.702 INFO:tasks.workunit.client.0.trial051.stdout:Complete! 2026-02-24T16:44:09.731 INFO:tasks.workunit.client.0.trial051.stderr:+ sleep 10 2026-02-24T16:44:09.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:09 trial183 ceph-mon[43553]: pgmap v133: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.8 KiB/s wr, 1.07k op/s 2026-02-24T16:44:09.733 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2061900914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:09.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:09 trial151 ceph-mon[39453]: pgmap v133: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.8 KiB/s wr, 1.07k op/s 2026-02-24T16:44:09.734 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2061900914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:09.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:09 trial153 ceph-mon[44017]: pgmap v133: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 5.8 KiB/s wr, 1.07k op/s 2026-02-24T16:44:09.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2061900914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:11 trial183 ceph-mon[43553]: pgmap v134: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 995 KiB/s rd, 4.2 KiB/s wr, 911 op/s 2026-02-24T16:44:11.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:11 trial151 ceph-mon[39453]: pgmap v134: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 995 KiB/s rd, 4.2 KiB/s wr, 911 op/s 2026-02-24T16:44:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:11 trial153 ceph-mon[44017]: pgmap v134: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 995 KiB/s rd, 4.2 KiB/s wr, 911 op/s 2026-02-24T16:44:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:13 trial183 ceph-mon[43553]: pgmap v135: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 990 KiB/s rd, 4.2 KiB/s wr, 882 op/s 2026-02-24T16:44:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3861664142' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4002932919' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/729574128' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:13 trial151 ceph-mon[39453]: pgmap v135: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 990 KiB/s rd, 4.2 KiB/s wr, 882 op/s 2026-02-24T16:44:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3861664142' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4002932919' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/729574128' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:13 trial153 ceph-mon[44017]: pgmap v135: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 990 KiB/s rd, 4.2 KiB/s wr, 882 op/s 2026-02-24T16:44:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3861664142' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4002932919' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/729574128' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:15 trial183 ceph-mon[43553]: pgmap v136: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:15.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:15 trial151 ceph-mon[39453]: pgmap v136: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:16.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:15 trial153 ceph-mon[44017]: pgmap v136: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/491760735' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:16.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/491760735' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:17.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/491760735' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:17.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:17 trial183 ceph-mon[43553]: pgmap v137: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:17.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:17 trial151 ceph-mon[39453]: pgmap v137: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:18.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:17 trial153 ceph-mon[44017]: pgmap v137: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 549 KiB/s rd, 2.0 KiB/s wr, 499 op/s 2026-02-24T16:44:19.733 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo modprobe nvme-fabrics 2026-02-24T16:44:19.750 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo modprobe nvme-tcp 2026-02-24T16:44:19.764 INFO:tasks.workunit.client.0.trial051.stderr:+ nvme version 2026-02-24T16:44:19.765 INFO:tasks.workunit.client.0.trial051.stdout:nvme version 2.13 (git 2.13) 2026-02-24T16:44:19.765 INFO:tasks.workunit.client.0.trial051.stdout:libnvme version 1.13 (git 1.13) 2026-02-24T16:44:19.765 INFO:tasks.workunit.client.0.trial051.stderr:+ sleep 20 2026-02-24T16:44:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:19 trial183 ceph-mon[43553]: pgmap v138: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 327 KiB/s rd, 1.2 KiB/s wr, 205 op/s 2026-02-24T16:44:19.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:19 trial151 ceph-mon[39453]: pgmap v138: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 327 KiB/s rd, 1.2 KiB/s wr, 205 op/s 2026-02-24T16:44:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:19 trial153 ceph-mon[44017]: pgmap v138: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 327 KiB/s rd, 1.2 KiB/s wr, 205 op/s 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1605900283' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2401947802' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2669996868' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1605900283' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2401947802' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:20.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2669996868' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:21.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1605900283' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2401947802' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2669996868' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:21.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:21 trial183 ceph-mon[43553]: pgmap v139: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 77 KiB/s rd, 0 B/s wr, 43 op/s 2026-02-24T16:44:21.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:21 trial151 ceph-mon[39453]: pgmap v139: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 77 KiB/s rd, 0 B/s wr, 43 op/s 2026-02-24T16:44:22.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:21 trial153 ceph-mon[44017]: pgmap v139: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 77 KiB/s rd, 0 B/s wr, 43 op/s 2026-02-24T16:44:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:23 trial183 ceph-mon[43553]: pgmap v140: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 0 B/s wr, 16 op/s 2026-02-24T16:44:23.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2072115616' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:23 trial151 ceph-mon[39453]: pgmap v140: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 0 B/s wr, 16 op/s 2026-02-24T16:44:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2072115616' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:23 trial153 ceph-mon[44017]: pgmap v140: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s rd, 0 B/s wr, 16 op/s 2026-02-24T16:44:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2072115616' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:25 trial183 ceph-mon[43553]: pgmap v141: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:25.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:25 trial151 ceph-mon[39453]: pgmap v141: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:26.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:25 trial153 ceph-mon[44017]: pgmap v141: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:27 trial183 ceph-mon[43553]: pgmap v142: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2629167278' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:27.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2712350138' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:27.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2913986511' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:27 trial151 ceph-mon[39453]: pgmap v142: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2629167278' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2712350138' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:27.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2913986511' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:27 trial153 ceph-mon[44017]: pgmap v142: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2629167278' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2712350138' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2913986511' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:29 trial183 ceph-mon[43553]: pgmap v143: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:29 trial151 ceph-mon[39453]: pgmap v143: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:29 trial153 ceph-mon[44017]: pgmap v143: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1822222909' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:30.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1822222909' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:31.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1822222909' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:31.884 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:31 trial183 ceph-mon[43553]: pgmap v144: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:31.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:31 trial151 ceph-mon[39453]: pgmap v144: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:32.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:31 trial153 ceph-mon[44017]: pgmap v144: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:33 trial183 ceph-mon[43553]: pgmap v145: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:33 trial151 ceph-mon[39453]: pgmap v145: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:33 trial153 ceph-mon[44017]: pgmap v145: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3427272392' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/876017722' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:34.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1904552980' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3427272392' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/876017722' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:34.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1904552980' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:35.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3427272392' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/876017722' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1904552980' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:35.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:35 trial183 ceph-mon[43553]: pgmap v146: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:35.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:35 trial151 ceph-mon[39453]: pgmap v146: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:36.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:35 trial153 ceph-mon[44017]: pgmap v146: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:37 trial183 ceph-mon[43553]: pgmap v147: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3697403441' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:37.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:37 trial151 ceph-mon[39453]: pgmap v147: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:37.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3697403441' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:38.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:37 trial153 ceph-mon[44017]: pgmap v147: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:38.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3697403441' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:39.766 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo lsmod 2026-02-24T16:44:39.767 INFO:tasks.workunit.client.0.trial051.stderr:+ grep nvme 2026-02-24T16:44:39.779 INFO:tasks.workunit.client.0.trial051.stdout:nvme_tcp 102400 0 2026-02-24T16:44:39.779 INFO:tasks.workunit.client.0.trial051.stdout:nvme_fabrics 49152 1 nvme_tcp 2026-02-24T16:44:39.780 INFO:tasks.workunit.client.0.trial051.stdout:nvme 73728 7 2026-02-24T16:44:39.780 INFO:tasks.workunit.client.0.trial051.stdout:nvme_core 274432 10 nvme_tcp,nvme,nvme_fabrics 2026-02-24T16:44:39.780 INFO:tasks.workunit.client.0.trial051.stdout:nvme_keyring 20480 3 nvme_tcp,nvme_core,nvme_fabrics 2026-02-24T16:44:39.780 INFO:tasks.workunit.client.0.trial051.stdout:nvme_auth 36864 1 nvme_core 2026-02-24T16:44:39.781 INFO:tasks.workunit.client.0.trial051.stderr:+ source /etc/ceph/nvmeof.env 2026-02-24T16:44:39.781 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:44:39.781 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_GATEWAY_NAMES=trial151,trial153,trial183,trial191 2026-02-24T16:44:39.781 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=10.20.193.151 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:devel 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_PORT=4420 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:++ NVMEOF_SRPORT=5500 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:+ SPDK_CONTROLLER='Ceph bdev Controller' 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:+ DISCOVERY_PORT=8009 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run disconnect_all 2026-02-24T16:44:39.782 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2026-02-24T16:44:39.783 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: disconnect_all 2026-02-24T16:44:39.783 INFO:tasks.workunit.client.0.trial051.stderr:+ disconnect_all 2026-02-24T16:44:39.783 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo nvme disconnect-all 2026-02-24T16:44:39.790 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme0n1 /dev/ng0n1 25174FA02D54 Micron_7450_MTFDKBA960TFR 0x1 25.59 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47B0F MTFDKCC1T9TGP-1BK1JABYY 0x1 144.22 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47B13 MTFDKCC1T9TGP-1BK1JABYY 0x1 138.70 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:44:39.802 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme0n1 /dev/ng0n1 25174FA02D54 Micron_7450_MTFDKBA960TFR 0x1 25.59 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47B0F MTFDKCC1T9TGP-1BK1JABYY 0x1 144.22 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47B13 MTFDKCC1T9TGP-1BK1JABYY 0x1 138.70 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:44:39.803 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:39.804 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] disconnect_all test passed! 2026-02-24T16:44:39.804 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2026-02-24T16:44:39.804 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run discovery 2026-02-24T16:44:39.805 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: discovery 2026-02-24T16:44:39.805 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: discovery' 2026-02-24T16:44:39.805 INFO:tasks.workunit.client.0.trial051.stderr:+ discovery 2026-02-24T16:44:39.806 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme discover -t tcp -a 10.20.193.151 -s 8009 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:+ output=' 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:Discovery Log Number of Records 12, Generation counter 1 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 0====== 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:portid: 0 2026-02-24T16:44:39.865 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 1====== 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:portid: 1 2026-02-24T16:44:39.866 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 2====== 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:portid: 2 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:39.867 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 3====== 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:portid: 3 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:39.868 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 4====== 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:portid: 4 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 5====== 2026-02-24T16:44:39.869 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:portid: 5 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 6====== 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.870 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:portid: 6 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 7====== 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.871 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:portid: 7 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 8====== 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:portid: 8 2026-02-24T16:44:39.872 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 9====== 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:portid: 9 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:39.873 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 10====== 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:portid: 10 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:39.874 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 11====== 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:portid: 11 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none' 2026-02-24T16:44:39.875 INFO:tasks.workunit.client.0.trial051.stderr:+ sleep 5 2026-02-24T16:44:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:39 trial183 ceph-mon[43553]: pgmap v148: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:39.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-mon[39453]: pgmap v148: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:39.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:39] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:39.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:39] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:39.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:39] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:39.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:39] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:39.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:39 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:39] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:40.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:39 trial153 ceph-mon[44017]: pgmap v148: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:44:40.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2664287171' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2152727473' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/733348028' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4058860074' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3370147352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2664287171' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2152727473' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/733348028' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4058860074' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:40.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3370147352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2664287171' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2152727473' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/733348028' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4058860074' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3370147352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:41 trial183 ceph-mon[43553]: pgmap v149: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3267568613' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2819761374' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/349314364' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:41 trial151 ceph-mon[39453]: pgmap v149: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3267568613' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2819761374' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/349314364' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:42.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:41 trial153 ceph-mon[44017]: pgmap v149: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:44:42.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3267568613' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2819761374' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/349314364' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:43.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:43 trial183 ceph-mon[43553]: pgmap v150: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:44:43.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:43 trial151 ceph-mon[39453]: pgmap v150: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:44:44.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:43 trial153 ceph-mon[44017]: pgmap v150: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:44:44.866 INFO:tasks.workunit.client.0.trial051.stderr:+ expected_discovery_stdout='subtype: nvme subsystem' 2026-02-24T16:44:44.866 INFO:tasks.workunit.client.0.trial051.stderr:+ echo ' 2026-02-24T16:44:44.866 INFO:tasks.workunit.client.0.trial051.stderr:Discovery Log Number of Records 12, Generation counter 1 2026-02-24T16:44:44.866 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 0====== 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:portid: 0 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 1====== 2026-02-24T16:44:44.867 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:portid: 1 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 2====== 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.868 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:portid: 2 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 3====== 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.869 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:portid: 3 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 4====== 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.870 INFO:tasks.workunit.client.0.trial051.stderr:portid: 4 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 5====== 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:portid: 5 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.871 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 6====== 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:portid: 6 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:44.872 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 7====== 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:portid: 7 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.873 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 8====== 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:portid: 8 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.151 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 9====== 2026-02-24T16:44:44.874 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:portid: 9 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.153 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 10====== 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.875 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:portid: 10 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.183 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:=====Discovery Log Entry 11====== 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:trtype: tcp 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:adrfam: ipv4 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:subtype: nvme subsystem 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:treq: not required 2026-02-24T16:44:44.876 INFO:tasks.workunit.client.0.trial051.stderr:portid: 11 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:trsvcid: 4420 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:traddr: 10.20.193.191 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:eflags: none 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:sectype: none' 2026-02-24T16:44:44.877 INFO:tasks.workunit.client.0.trial051.stderr:+ grep -q 'subtype: nvme subsystem' 2026-02-24T16:44:44.878 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:44.878 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] discovery test passed!' 2026-02-24T16:44:44.878 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] discovery test passed! 2026-02-24T16:44:44.879 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run connect 2026-02-24T16:44:44.879 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: connect' 2026-02-24T16:44:44.879 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: connect 2026-02-24T16:44:44.879 INFO:tasks.workunit.client.0.trial051.stderr:+ connect 2026-02-24T16:44:44.880 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo nvme connect -t tcp --traddr 10.20.193.151 -s 4420 -n nqn.2016-06.io.spdk:cnode1 2026-02-24T16:44:44.958 INFO:tasks.workunit.client.0.trial051.stdout:connecting to device: nvme3 2026-02-24T16:44:44.960 INFO:tasks.workunit.client.0.trial051.stderr:+ sleep 5 2026-02-24T16:44:44.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2549107588' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:44.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2549107588' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:45.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2549107588' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:45 trial183 ceph-mon[43553]: pgmap v151: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:44:45.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:45 trial151 ceph-mon[39453]: pgmap v151: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:44:46.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:45 trial153 ceph-mon[44017]: pgmap v151: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:44:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:47 trial183 ceph-mon[43553]: pgmap v152: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:44:47.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:47 trial151 ceph-mon[39453]: pgmap v152: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:44:48.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:47 trial153 ceph-mon[44017]: pgmap v152: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:44:48.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2660436664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:48.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2775893175' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:48.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1555986156' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2660436664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2775893175' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1555986156' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:49.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2660436664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:49.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2775893175' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:49.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1555986156' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:49.899 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:49 trial153 ceph-mon[44017]: pgmap v153: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:44:49.901 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:49 trial183 ceph-mon[43553]: pgmap v153: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:44:49.937 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:49 trial151 ceph-mon[39453]: pgmap v153: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:44:49.962 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list --output-format=json 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr:+ output='{ 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "Devices":[ 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme0n1", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng0n1", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E2MU200", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Micron_7450_MTFDKBA960TFR", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"25174FA02D54", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":25586667520, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":1875385008, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":960197124096, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme1n1", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng1n1", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E3MQ000", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"012510B47B0F", 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":144220020736, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":3750748848, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:44:49.977 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme2n1", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng2n1", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E3MQ000", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"012510B47B13", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":138703310848, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":3750748848, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":12, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n12", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n12", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":16, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n16", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n16", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":20, 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n20", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n20", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.978 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":4, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n4", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n4", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":8, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n8", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n8", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr:}' 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '{ 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "Devices":[ 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme0n1", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng0n1", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E2MU200", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Micron_7450_MTFDKBA960TFR", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"25174FA02D54", 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":25586667520, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":1875385008, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":960197124096, 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.979 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme1n1", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng1n1", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E3MQ000", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"012510B47B0F", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":144220020736, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":3750748848, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":1, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme2n1", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng2n1", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"E3MQ000", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"012510B47B13", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":138703310848, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":3750748848, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":12, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n12", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n12", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":16, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n16", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n16", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.980 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":20, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n20", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n20", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":4, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n4", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n4", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "NameSpace":8, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "DevicePath":"/dev/nvme3n8", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "GenericPath":"/dev/ng3n8", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "Firmware":"25.09", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "UsedBytes":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "MaximumLBA":16777216, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "PhysicalSize":8589934592, 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: "SectorSize":512 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.981 INFO:tasks.workunit.client.0.trial051.stderr:}' 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] connect test passed! 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] connect test passed!' 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run list_subsys 1 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ list_subsys 1 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:+ expected_count=1 2026-02-24T16:44:49.982 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr:+ output='[ 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Subsystems":[ 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys0", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme0", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:02:00.0", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys1", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme1", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:03:00.0", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys2", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme2", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:04:00.0", 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:49.991 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys3", 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme3", 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr:]' 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys3",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"Paths":[' '{' '"Name":"nvme3",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr:++ grep -o '"tcp"' 2026-02-24T16:44:49.992 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -l 2026-02-24T16:44:49.993 INFO:tasks.workunit.client.0.trial051.stderr:+ multipath=1 2026-02-24T16:44:49.994 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 1 -ne 1 ']' 2026-02-24T16:44:49.994 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:49.994 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:44:49.994 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:44:49.994 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run disconnect_all 2026-02-24T16:44:49.995 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2026-02-24T16:44:49.995 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: disconnect_all 2026-02-24T16:44:49.995 INFO:tasks.workunit.client.0.trial051.stderr:+ disconnect_all 2026-02-24T16:44:49.995 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo nvme disconnect-all 2026-02-24T16:44:50.354 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme0n1 /dev/ng0n1 25174FA02D54 Micron_7450_MTFDKBA960TFR 0x1 25.59 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47B0F MTFDKCC1T9TGP-1BK1JABYY 0x1 144.22 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47B13 MTFDKCC1T9TGP-1BK1JABYY 0x1 138.70 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:44:50.366 INFO:tasks.workunit.client.0.trial051.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:44:50.367 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme0n1 /dev/ng0n1 25174FA02D54 Micron_7450_MTFDKBA960TFR 0x1 25.59 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:44:50.367 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47B0F MTFDKCC1T9TGP-1BK1JABYY 0x1 144.22 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:44:50.367 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47B13 MTFDKCC1T9TGP-1BK1JABYY 0x1 138.70 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:44:50.367 INFO:tasks.workunit.client.0.trial051.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:44:50.368 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:50.368 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] disconnect_all test passed! 2026-02-24T16:44:50.369 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2026-02-24T16:44:50.369 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run list_subsys 0 2026-02-24T16:44:50.369 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:44:50.370 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:44:50.370 INFO:tasks.workunit.client.0.trial051.stderr:+ list_subsys 0 2026-02-24T16:44:50.370 INFO:tasks.workunit.client.0.trial051.stderr:+ expected_count=0 2026-02-24T16:44:50.370 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:44:50.377 INFO:tasks.workunit.client.0.trial051.stderr:+ output='[ 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "Subsystems":[ 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys0", 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54", 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:50.378 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme0", 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:02:00.0", 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys1", 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F", 2026-02-24T16:44:50.379 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme1", 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:03:00.0", 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.380 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys2", 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13", 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme2", 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:04:00.0", 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:50.381 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:50.382 INFO:tasks.workunit.client.0.trial051.stderr:]' 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:++ grep -o '"tcp"' 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -l 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:+ multipath=0 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -ne 0 ']' 2026-02-24T16:44:50.383 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:50.384 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:44:50.384 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:44:50.384 INFO:tasks.workunit.client.0.trial051.stderr:+ devices_count=60 2026-02-24T16:44:50.385 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run connect_all 60 2026-02-24T16:44:50.385 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: connect_all 2026-02-24T16:44:50.385 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: connect_all' 2026-02-24T16:44:50.385 INFO:tasks.workunit.client.0.trial051.stderr:+ connect_all 60 2026-02-24T16:44:50.386 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo nvme connect-all --traddr=10.20.193.151 --transport=tcp -l 3600 2026-02-24T16:44:50.649 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:50] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:50.649 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:50] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:50.649 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:50] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:50.649 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:50] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:50.649 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:44:50] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:44:50.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3353875363' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4018867685' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1845343565' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/107930670' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3481447914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:44:50.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3353875363' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4018867685' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1845343565' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/107930670' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:50.988 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3481447914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3353875363' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4018867685' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1845343565' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/107930670' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3481447914' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.540 INFO:tasks.workunit.client.0.trial051.stderr:+ sleep 5 2026-02-24T16:44:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:51 trial183 ceph-mon[43553]: pgmap v154: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:44:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1777888798' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:51.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:51 trial151 ceph-mon[39453]: pgmap v154: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:44:51.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1777888798' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:52.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:51 trial153 ceph-mon[44017]: pgmap v154: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:44:52.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1777888798' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:53.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:53 trial183 ceph-mon[43553]: pgmap v155: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 1.6 KiB/s wr, 20 op/s 2026-02-24T16:44:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:53 trial151 ceph-mon[39453]: pgmap v155: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 1.6 KiB/s wr, 20 op/s 2026-02-24T16:44:54.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:53 trial153 ceph-mon[44017]: pgmap v155: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 1.6 KiB/s wr, 20 op/s 2026-02-24T16:44:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:55 trial183 ceph-mon[43553]: pgmap v156: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2026-02-24T16:44:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3423917060' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:55.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3330792092' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:55.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1000601107' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:55 trial151 ceph-mon[39453]: pgmap v156: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2026-02-24T16:44:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3423917060' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3330792092' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:55.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1000601107' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:55 trial153 ceph-mon[44017]: pgmap v156: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2026-02-24T16:44:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3423917060' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3330792092' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1000601107' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:56.541 INFO:tasks.workunit.client.0.trial051.stderr:+ expected_devices_count=60 2026-02-24T16:44:56.542 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list --output-format=json 2026-02-24T16:44:56.542 INFO:tasks.workunit.client.0.trial051.stderr:++ grep -o 'Ceph bdev Controller' 2026-02-24T16:44:56.543 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -l 2026-02-24T16:44:56.575 INFO:tasks.workunit.client.0.trial051.stderr:+ actual_devices=60 2026-02-24T16:44:56.575 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 60 -ne 60 ']' 2026-02-24T16:44:56.576 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:56.576 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] connect_all test passed! 2026-02-24T16:44:56.577 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] connect_all test passed!' 2026-02-24T16:44:56.577 INFO:tasks.workunit.client.0.trial051.stderr:++ echo 10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:44:56.577 INFO:tasks.workunit.client.0.trial051.stderr:++ tr -cd , 2026-02-24T16:44:56.577 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -c 2026-02-24T16:44:56.577 INFO:tasks.workunit.client.0.trial051.stderr:+ gateways_count=4 2026-02-24T16:44:56.578 INFO:tasks.workunit.client.0.trial051.stderr:+ multipath_count=12 2026-02-24T16:44:56.578 INFO:tasks.workunit.client.0.trial051.stderr:+ test_run list_subsys 12 2026-02-24T16:44:56.578 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:44:56.579 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:44:56.579 INFO:tasks.workunit.client.0.trial051.stderr:+ list_subsys 12 2026-02-24T16:44:56.579 INFO:tasks.workunit.client.0.trial051.stderr:+ expected_count=12 2026-02-24T16:44:56.579 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr:+ output='[ 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "Subsystems":[ 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys0", 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54", 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme0", 2026-02-24T16:44:56.598 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:02:00.0", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys1", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme1", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:03:00.0", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys8", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme10", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.599 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme11", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme8", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme9", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys12", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme12", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.600 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme13", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme14", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme15", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys2", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13", 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.601 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme2", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"pcie", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"0000:04:00.0", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme-subsys4", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Paths":[ 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme4", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme5", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme6", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: }, 2026-02-24T16:44:56.602 INFO:tasks.workunit.client.0.trial051.stderr: { 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: "Name":"nvme7", 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: "Transport":"tcp", 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51", 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: "State":"live" 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: ] 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr: } 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr:]' 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FA02D54",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B0F",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys8",' '"NQN":"nqn.2016-06.io.spdk:cnode2",' '"Paths":[' '{' '"Name":"nvme10",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme11",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme8",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme9",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys12",' '"NQN":"nqn.2016-06.io.spdk:cnode3",' '"Paths":[' '{' '"Name":"nvme12",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme13",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme14",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme15",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47B13",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys4",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"Paths":[' '{' '"Name":"nvme4",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme5",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme6",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '},' '{' '"Name":"nvme7",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.51",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:44:56.603 INFO:tasks.workunit.client.0.trial051.stderr:++ grep -o '"tcp"' 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:++ wc -l 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:+ multipath=12 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 12 -ne 12 ']' 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stdout:-------------Test Summary------------- 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '-------------Test Summary-------------' 2026-02-24T16:44:56.604 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof] All nvmeof basic tests passed! 2026-02-24T16:44:56.605 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof] All nvmeof basic tests passed!' 2026-02-24T16:44:56.605 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T16:44:56.605 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-24T16:44:56.670 INFO:tasks.workunit:Stopping ['nvmeof/setup_subsystem.sh', 'nvmeof/basic_tests.sh'] on client.0... 2026-02-24T16:44:56.670 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-24T16:44:57.027 DEBUG:teuthology.parallel:result is None 2026-02-24T16:44:57.027 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:44:57.051 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:44:57.051 DEBUG:teuthology.orchestra.run.trial051:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:44:57.108 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:44:57.108 INFO:teuthology.run_tasks:Running task workunit... 2026-02-24T16:44:57.121 INFO:tasks.workunit:Pulling workunits from ref 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:44:57.121 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-24T16:44:57.121 DEBUG:teuthology.orchestra.run.trial051:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:44:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:44:57.165 INFO:teuthology.orchestra.run.trial051.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-24T16:44:57.165 DEBUG:teuthology.orchestra.run.trial051:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:44:57.220 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-24T16:44:57.221 DEBUG:teuthology.orchestra.run.trial051:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-24T16:44:57.277 DEBUG:teuthology.orchestra.run.trial052:> stat -- /home/ubuntu/cephtest/mnt.1 2026-02-24T16:44:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:44:57.307 INFO:teuthology.orchestra.run.trial052.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.1': No such file or directory 2026-02-24T16:44:57.307 DEBUG:teuthology.orchestra.run.trial052:> mkdir -- /home/ubuntu/cephtest/mnt.1 2026-02-24T16:44:57.363 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.1 2026-02-24T16:44:57.364 DEBUG:teuthology.orchestra.run.trial052:> cd -- /home/ubuntu/cephtest/mnt.1 && mkdir -- client.1 2026-02-24T16:44:57.419 INFO:tasks.workunit:timeout=30m 2026-02-24T16:44:57.419 INFO:tasks.workunit:cleanup=True 2026-02-24T16:44:57.419 DEBUG:teuthology.orchestra.run.trial051:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:44:57.420 DEBUG:teuthology.orchestra.run.trial052:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T16:44:57.438 INFO:tasks.workunit.client.0.trial051.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-24T16:44:57.482 INFO:tasks.workunit.client.1.trial052.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2026-02-24T16:44:57.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:57 trial183 ceph-mon[43553]: pgmap v157: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.2 KiB/s wr, 16 op/s 2026-02-24T16:44:57.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:57 trial151 ceph-mon[39453]: pgmap v157: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.2 KiB/s wr, 16 op/s 2026-02-24T16:44:58.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:57 trial153 ceph-mon[44017]: pgmap v157: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.2 KiB/s wr, 16 op/s 2026-02-24T16:44:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/127114461' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:58.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/127114461' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:59.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/127114461' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:44:59.964 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:44:59 trial153 ceph-mon[44017]: pgmap v158: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.9 KiB/s wr, 20 op/s 2026-02-24T16:44:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:44:59 trial183 ceph-mon[43553]: pgmap v158: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.9 KiB/s wr, 20 op/s 2026-02-24T16:44:59.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:44:59 trial151 ceph-mon[39453]: pgmap v158: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.9 KiB/s wr, 20 op/s 2026-02-24T16:45:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:01 trial183 ceph-mon[43553]: pgmap v159: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2026-02-24T16:45:01.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:01 trial151 ceph-mon[39453]: pgmap v159: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2026-02-24T16:45:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:01 trial153 ceph-mon[44017]: pgmap v159: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2026-02-24T16:45:02.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4251487202' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:02.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/894472275' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:02.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4173245086' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4251487202' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/894472275' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4173245086' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4251487202' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/894472275' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4173245086' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:03.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:03 trial183 ceph-mon[43553]: pgmap v160: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.8 KiB/s wr, 20 op/s 2026-02-24T16:45:03.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:03 trial151 ceph-mon[39453]: pgmap v160: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.8 KiB/s wr, 20 op/s 2026-02-24T16:45:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:03 trial153 ceph-mon[44017]: pgmap v160: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 2.8 KiB/s wr, 20 op/s 2026-02-24T16:45:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:05 trial183 ceph-mon[43553]: pgmap v161: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2837080239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:05.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:05 trial151 ceph-mon[39453]: pgmap v161: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:05.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2837080239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:05 trial153 ceph-mon[44017]: pgmap v161: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2837080239' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:07 trial183 ceph-mon[43553]: pgmap v162: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:07.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:07 trial151 ceph-mon[39453]: pgmap v162: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:08.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:07 trial153 ceph-mon[44017]: pgmap v162: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2026-02-24T16:45:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:09 trial183 ceph-mon[43553]: pgmap v163: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4146495664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3138777699' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/201592111' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:09 trial151 ceph-mon[39453]: pgmap v163: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4146495664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3138777699' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:09.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/201592111' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:09 trial153 ceph-mon[44017]: pgmap v163: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4146495664' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3138777699' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/201592111' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:11 trial183 ceph-mon[43553]: pgmap v164: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:11.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:11 trial151 ceph-mon[39453]: pgmap v164: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:12.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:11 trial153 ceph-mon[44017]: pgmap v164: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:12.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2728148303' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:12.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2728148303' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:13.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2728148303' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:13 trial183 ceph-mon[43553]: pgmap v165: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:13 trial151 ceph-mon[39453]: pgmap v165: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:13 trial153 ceph-mon[44017]: pgmap v165: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:15 trial183 ceph-mon[43553]: pgmap v166: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:15.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:15 trial151 ceph-mon[39453]: pgmap v166: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:16.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:15 trial153 ceph-mon[44017]: pgmap v166: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/97747346' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:16.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3702348931' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:16.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3978848450' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:16.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/97747346' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:16.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3702348931' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:16.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3978848450' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:17.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/97747346' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3702348931' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3978848450' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:17.525 INFO:tasks.workunit.client.1.trial052.stderr:Note: switching to '618c318a7a0b287ab5aeff207b7bf1d73ef45044'. 2026-02-24T16:45:17.525 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.525 INFO:tasks.workunit.client.1.trial052.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-24T16:45:17.525 INFO:tasks.workunit.client.1.trial052.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr:state without impacting any branches by switching back to a branch. 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr: git switch -c 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr:Or undo this operation with: 2026-02-24T16:45:17.526 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.527 INFO:tasks.workunit.client.1.trial052.stderr: git switch - 2026-02-24T16:45:17.527 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.527 INFO:tasks.workunit.client.1.trial052.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-24T16:45:17.527 INFO:tasks.workunit.client.1.trial052.stderr: 2026-02-24T16:45:17.527 INFO:tasks.workunit.client.1.trial052.stderr:HEAD is now at 618c318a7a0 Merge branch 'stretched-cluster' of https://github.com/leonidc/ceph into wip-yuri8-testing-2026-02-23-2128 2026-02-24T16:45:17.532 DEBUG:teuthology.orchestra.run.trial052:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2026-02-24T16:45:17.591 INFO:tasks.workunit.client.1.trial052.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-24T16:45:17.593 INFO:tasks.workunit.client.1.trial052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2026-02-24T16:45:17.593 INFO:tasks.workunit.client.1.trial052.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-24T16:45:17.689 INFO:tasks.workunit.client.1.trial052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-24T16:45:17.718 INFO:tasks.workunit.client.1.trial052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-24T16:45:17.743 INFO:tasks.workunit.client.1.trial052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2026-02-24T16:45:17.744 INFO:tasks.workunit.client.1.trial052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2026-02-24T16:45:17.744 INFO:tasks.workunit.client.1.trial052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-24T16:45:17.768 INFO:tasks.workunit.client.1.trial052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2026-02-24T16:45:17.772 DEBUG:teuthology.orchestra.run.trial052:> set -ex 2026-02-24T16:45:17.772 DEBUG:teuthology.orchestra.run.trial052:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2026-02-24T16:45:17.828 INFO:tasks.workunit:Running workunits matching nvmeof/basic_tests.sh on client.1... 2026-02-24T16:45:17.829 INFO:tasks.workunit:Running workunit nvmeof/basic_tests.sh... 2026-02-24T16:45:17.829 DEBUG:teuthology.orchestra.run.trial052:workunit test nvmeof/basic_tests.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 IOSTAT_INTERVAL=10 NEW_NAMESPACES_COUNT=5 RBD_POOL=mypool RUNTIME=120 timeout 30m /home/ubuntu/cephtest/clone.client.1/qa/workunits/nvmeof/basic_tests.sh 2026-02-24T16:45:17.886 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo systemctl stop udisks2 2026-02-24T16:45:17.900 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo dnf install nvme-cli-2.13 libnvme-1.13 -y 2026-02-24T16:45:17.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:17 trial183 ceph-mon[43553]: pgmap v167: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:17.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:17 trial151 ceph-mon[39453]: pgmap v167: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:18.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:17 trial153 ceph-mon[44017]: pgmap v167: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:18.202 INFO:tasks.workunit.client.1.trial052.stdout:Last metadata expiration check: 0:07:28 ago on Tue 24 Feb 2026 04:37:50 PM UTC. 2026-02-24T16:45:18.286 INFO:tasks.workunit.client.1.trial052.stdout:Dependencies resolved. 2026-02-24T16:45:18.286 INFO:tasks.workunit.client.1.trial052.stdout:================================================================================ 2026-02-24T16:45:18.286 INFO:tasks.workunit.client.1.trial052.stdout: Package Architecture Version Repository Size 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:================================================================================ 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:Downgrading: 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout: libnvme x86_64 1.13-1.el9 baseos 111 k 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout: 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:Transaction Summary 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:================================================================================ 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:Downgrade 2 Packages 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout: 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:Total download size: 1.2 M 2026-02-24T16:45:18.287 INFO:tasks.workunit.client.1.trial052.stdout:Downloading Packages: 2026-02-24T16:45:18.460 INFO:tasks.workunit.client.1.trial052.stdout:(1/2): libnvme-1.13-1.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2026-02-24T16:45:18.563 INFO:tasks.workunit.client.1.trial052.stdout:(2/2): nvme-cli-2.13-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2026-02-24T16:45:18.564 INFO:tasks.workunit.client.1.trial052.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:45:18.564 INFO:tasks.workunit.client.1.trial052.stdout:Total 4.4 MB/s | 1.2 MB 00:00 2026-02-24T16:45:18.582 INFO:tasks.workunit.client.1.trial052.stdout:Running transaction check 2026-02-24T16:45:18.588 INFO:tasks.workunit.client.1.trial052.stdout:Transaction check succeeded. 2026-02-24T16:45:18.588 INFO:tasks.workunit.client.1.trial052.stdout:Running transaction test 2026-02-24T16:45:18.632 INFO:tasks.workunit.client.1.trial052.stdout:Transaction test succeeded. 2026-02-24T16:45:18.632 INFO:tasks.workunit.client.1.trial052.stdout:Running transaction 2026-02-24T16:45:18.697 INFO:tasks.workunit.client.1.trial052.stdout: Preparing : 1/1 2026-02-24T16:45:18.818 INFO:tasks.workunit.client.1.trial052.stdout: Downgrading : libnvme-1.13-1.el9.x86_64 1/4 2026-02-24T16:45:18.820 INFO:tasks.workunit.client.1.trial052.stdout: Downgrading : nvme-cli-2.13-1.el9.x86_64 2/4 2026-02-24T16:45:19.010 INFO:tasks.workunit.client.1.trial052.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 2/4 2026-02-24T16:45:19.014 INFO:tasks.workunit.client.1.trial052.stdout: Cleanup : nvme-cli-2.16-1.el9.x86_64 3/4 2026-02-24T16:45:19.038 INFO:tasks.workunit.client.1.trial052.stdout: Cleanup : libnvme-1.16.1-2.el9.x86_64 4/4 2026-02-24T16:45:19.484 INFO:tasks.workunit.client.1.trial052.stdout: Running scriptlet: libnvme-1.16.1-2.el9.x86_64 4/4 2026-02-24T16:45:19.484 INFO:tasks.workunit.client.1.trial052.stdout: Verifying : libnvme-1.13-1.el9.x86_64 1/4 2026-02-24T16:45:19.484 INFO:tasks.workunit.client.1.trial052.stdout: Verifying : libnvme-1.16.1-2.el9.x86_64 2/4 2026-02-24T16:45:19.484 INFO:tasks.workunit.client.1.trial052.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 3/4 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 4/4 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout: 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout:Downgraded: 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout: libnvme-1.13-1.el9.x86_64 nvme-cli-2.13-1.el9.x86_64 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout: 2026-02-24T16:45:19.548 INFO:tasks.workunit.client.1.trial052.stdout:Complete! 2026-02-24T16:45:19.575 INFO:tasks.workunit.client.1.trial052.stderr:+ sleep 10 2026-02-24T16:45:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:19 trial183 ceph-mon[43553]: pgmap v168: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:19 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2777738680' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:19.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:19 trial151 ceph-mon[39453]: pgmap v168: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:19.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2777738680' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:19 trial153 ceph-mon[44017]: pgmap v168: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:19 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2777738680' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:21.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:21 trial183 ceph-mon[43553]: pgmap v169: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:21.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:21 trial151 ceph-mon[39453]: pgmap v169: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:22.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:21 trial153 ceph-mon[44017]: pgmap v169: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:23 trial183 ceph-mon[43553]: pgmap v170: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/630783189' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/55585617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/556083871' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:23 trial151 ceph-mon[39453]: pgmap v170: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/630783189' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/55585617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/556083871' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:23 trial153 ceph-mon[44017]: pgmap v170: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/630783189' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/55585617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/556083871' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:25 trial183 ceph-mon[43553]: pgmap v171: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:25.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:25 trial151 ceph-mon[39453]: pgmap v171: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:26.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:25 trial153 ceph-mon[44017]: pgmap v171: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:26.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1913641178' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:26.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1913641178' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:27.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1913641178' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:27 trial183 ceph-mon[43553]: pgmap v172: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:27 trial151 ceph-mon[39453]: pgmap v172: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:28.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:27 trial153 ceph-mon[44017]: pgmap v172: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:29.576 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo modprobe nvme-fabrics 2026-02-24T16:45:29.594 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo modprobe nvme-tcp 2026-02-24T16:45:29.608 INFO:tasks.workunit.client.1.trial052.stderr:+ nvme version 2026-02-24T16:45:29.609 INFO:tasks.workunit.client.1.trial052.stdout:nvme version 2.13 (git 2.13) 2026-02-24T16:45:29.610 INFO:tasks.workunit.client.1.trial052.stdout:libnvme version 1.13 (git 1.13) 2026-02-24T16:45:29.610 INFO:tasks.workunit.client.1.trial052.stderr:+ sleep 20 2026-02-24T16:45:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:29 trial183 ceph-mon[43553]: pgmap v173: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:29.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:29 trial151 ceph-mon[39453]: pgmap v173: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:30.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:29 trial153 ceph-mon[44017]: pgmap v173: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3956191781' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2482726794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1781421076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:30.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3956191781' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:30.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2482726794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:30.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1781421076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:31.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3956191781' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2482726794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1781421076' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:31.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:31 trial183 ceph-mon[43553]: pgmap v174: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:31.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:31 trial151 ceph-mon[39453]: pgmap v174: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:32.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:31 trial153 ceph-mon[44017]: pgmap v174: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:33 trial183 ceph-mon[43553]: pgmap v175: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:33 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/974811987' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:33 trial151 ceph-mon[39453]: pgmap v175: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:33 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/974811987' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:33 trial153 ceph-mon[44017]: pgmap v175: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:34.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:33 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/974811987' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:35.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:35 trial183 ceph-mon[43553]: pgmap v176: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:35.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:35 trial151 ceph-mon[39453]: pgmap v176: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:36.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:35 trial153 ceph-mon[44017]: pgmap v176: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:37 trial183 ceph-mon[43553]: pgmap v177: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:37.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4116914192' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:37.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/869302657' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:37.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2468495434' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:37.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:37 trial151 ceph-mon[39453]: pgmap v177: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:37.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4116914192' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:37.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/869302657' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:37.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2468495434' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:38.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:37 trial153 ceph-mon[44017]: pgmap v177: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4116914192' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/869302657' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2468495434' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:39 trial183 ceph-mon[43553]: pgmap v178: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:39.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:39 trial151 ceph-mon[39453]: pgmap v178: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:40.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:39 trial153 ceph-mon[44017]: pgmap v178: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:40.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2602752749' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2602752749' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:41.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2602752749' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:41 trial183 ceph-mon[43553]: pgmap v179: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:41 trial151 ceph-mon[39453]: pgmap v179: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:42.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:41 trial153 ceph-mon[44017]: pgmap v179: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:43.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:43 trial183 ceph-mon[43553]: pgmap v180: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:43.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:43 trial151 ceph-mon[39453]: pgmap v180: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:44.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:43 trial153 ceph-mon[44017]: pgmap v180: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:44.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/812821979' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:44.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/824930953' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:44.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1603679604' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:44.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/812821979' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:44.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/824930953' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:44.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1603679604' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:45.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/812821979' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/824930953' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1603679604' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:45 trial183 ceph-mon[43553]: pgmap v181: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:45.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:45 trial151 ceph-mon[39453]: pgmap v181: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:46.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:45 trial153 ceph-mon[44017]: pgmap v181: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:47 trial183 ceph-mon[43553]: pgmap v182: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2661755858' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:47.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:47 trial151 ceph-mon[39453]: pgmap v182: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:47.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2661755858' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:48.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:47 trial153 ceph-mon[44017]: pgmap v182: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2661755858' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.611 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo lsmod 2026-02-24T16:45:49.611 INFO:tasks.workunit.client.1.trial052.stderr:+ grep nvme 2026-02-24T16:45:49.624 INFO:tasks.workunit.client.1.trial052.stdout:nvme_tcp 102400 0 2026-02-24T16:45:49.624 INFO:tasks.workunit.client.1.trial052.stdout:nvme_fabrics 49152 1 nvme_tcp 2026-02-24T16:45:49.624 INFO:tasks.workunit.client.1.trial052.stdout:nvme 73728 7 2026-02-24T16:45:49.625 INFO:tasks.workunit.client.1.trial052.stdout:nvme_core 274432 10 nvme_tcp,nvme,nvme_fabrics 2026-02-24T16:45:49.625 INFO:tasks.workunit.client.1.trial052.stdout:nvme_keyring 20480 3 nvme_tcp,nvme_core,nvme_fabrics 2026-02-24T16:45:49.625 INFO:tasks.workunit.client.1.trial052.stdout:nvme_auth 36864 1 nvme_core 2026-02-24T16:45:49.626 INFO:tasks.workunit.client.1.trial052.stderr:+ source /etc/ceph/nvmeof.env 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_GATEWAY_NAMES=trial151,trial153,trial183,trial191 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=10.20.193.151 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:devel 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_PORT=4420 2026-02-24T16:45:49.627 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SRPORT=5500 2026-02-24T16:45:49.628 INFO:tasks.workunit.client.1.trial052.stderr:+ SPDK_CONTROLLER='Ceph bdev Controller' 2026-02-24T16:45:49.628 INFO:tasks.workunit.client.1.trial052.stderr:+ DISCOVERY_PORT=8009 2026-02-24T16:45:49.628 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run disconnect_all 2026-02-24T16:45:49.628 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2026-02-24T16:45:49.628 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: disconnect_all 2026-02-24T16:45:49.629 INFO:tasks.workunit.client.1.trial052.stderr:+ disconnect_all 2026-02-24T16:45:49.629 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo nvme disconnect-all 2026-02-24T16:45:49.635 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list 2026-02-24T16:45:49.646 INFO:tasks.workunit.client.1.trial052.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:45:49.646 INFO:tasks.workunit.client.1.trial052.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:45:49.646 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme0n1 /dev/ng0n1 25174FAD8DBC Micron_7450_MTFDKBA960TFR 0x1 34.63 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:45:49.646 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47DC4 MTFDKCC1T9TGP-1BK1JABYY 0x1 157.30 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:45:49.646 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47A7F MTFDKCC1T9TGP-1BK1JABYY 0x1 147.88 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme0n1 /dev/ng0n1 25174FAD8DBC Micron_7450_MTFDKBA960TFR 0x1 34.63 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47DC4 MTFDKCC1T9TGP-1BK1JABYY 0x1 157.30 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47A7F MTFDKCC1T9TGP-1BK1JABYY 0x1 147.88 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:45:49.647 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:45:49.648 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] disconnect_all test passed! 2026-02-24T16:45:49.648 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2026-02-24T16:45:49.648 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run discovery 2026-02-24T16:45:49.649 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: discovery 2026-02-24T16:45:49.649 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: discovery' 2026-02-24T16:45:49.649 INFO:tasks.workunit.client.1.trial052.stderr:+ discovery 2026-02-24T16:45:49.650 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme discover -t tcp -a 10.20.193.151 -s 8009 2026-02-24T16:45:49.706 INFO:tasks.workunit.client.1.trial052.stderr:+ output=' 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:Discovery Log Number of Records 12, Generation counter 3 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 0====== 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:portid: 0 2026-02-24T16:45:49.707 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 1====== 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.708 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:portid: 1 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 2====== 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.709 INFO:tasks.workunit.client.1.trial052.stderr:portid: 2 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 3====== 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:portid: 3 2026-02-24T16:45:49.710 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 4====== 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:portid: 4 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.711 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 5====== 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:portid: 5 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.712 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 6====== 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:portid: 6 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 7====== 2026-02-24T16:45:49.713 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:portid: 7 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 8====== 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.714 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:portid: 8 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 9====== 2026-02-24T16:45:49.715 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:portid: 9 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 10====== 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.716 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:portid: 10 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 11====== 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:portid: 11 2026-02-24T16:45:49.717 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:49.718 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:49.718 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:49.718 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:49.718 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none' 2026-02-24T16:45:49.718 INFO:tasks.workunit.client.1.trial052.stderr:+ sleep 5 2026-02-24T16:45:49.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:49 trial183 ceph-mon[43553]: pgmap v183: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4291728657' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1721919414' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3954267874' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:49 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2874430372' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-mon[39453]: pgmap v183: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:49.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4291728657' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1721919414' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3954267874' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2874430372' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:49.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:45:49] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:45:49.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:45:49] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:45:49.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:45:49] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:45:49.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:45:49] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:45:49.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:45:49 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:45:49] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:45:50.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:49 trial153 ceph-mon[44017]: pgmap v183: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 71 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:45:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4291728657' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1721919414' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3954267874' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:49 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2874430372' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3987511095' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:50 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3987511095' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:45:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:50.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:50 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3987511095' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:45:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:50 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: pgmap v184: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2060117519' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2266905987' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1215296617' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: pgmap v184: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2060117519' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2266905987' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:51.987 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1215296617' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: pgmap v184: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2060117519' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2266905987' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:52.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1215296617' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:53.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:53 trial183 ceph-mon[43553]: pgmap v185: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:45:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:53 trial151 ceph-mon[39453]: pgmap v185: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:45:54.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:53 trial153 ceph-mon[44017]: pgmap v185: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 0 B/s wr, 14 op/s 2026-02-24T16:45:54.707 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_discovery_stdout='subtype: nvme subsystem' 2026-02-24T16:45:54.708 INFO:tasks.workunit.client.1.trial052.stderr:+ echo ' 2026-02-24T16:45:54.708 INFO:tasks.workunit.client.1.trial052.stderr:Discovery Log Number of Records 12, Generation counter 3 2026-02-24T16:45:54.708 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 0====== 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:portid: 0 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 1====== 2026-02-24T16:45:54.709 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:portid: 1 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 2====== 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.710 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:portid: 2 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 3====== 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.711 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:portid: 3 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 4====== 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.712 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:portid: 4 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 5====== 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:portid: 5 2026-02-24T16:45:54.713 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 6====== 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:portid: 6 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:54.714 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 7====== 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:portid: 7 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.715 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 8====== 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:portid: 8 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.151 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 9====== 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.716 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:portid: 9 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.153 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 10====== 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.717 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:portid: 10 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.183 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:=====Discovery Log Entry 11====== 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:trtype: tcp 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:adrfam: ipv4 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:subtype: nvme subsystem 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:treq: not required 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:portid: 11 2026-02-24T16:45:54.718 INFO:tasks.workunit.client.1.trial052.stderr:trsvcid: 4420 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:traddr: 10.20.193.191 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:eflags: none 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:sectype: none' 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:+ grep -q 'subtype: nvme subsystem' 2026-02-24T16:45:54.719 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:45:54.720 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] discovery test passed!' 2026-02-24T16:45:54.720 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] discovery test passed! 2026-02-24T16:45:54.720 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: connect 2026-02-24T16:45:54.721 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run connect 2026-02-24T16:45:54.721 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: connect' 2026-02-24T16:45:54.721 INFO:tasks.workunit.client.1.trial052.stderr:+ connect 2026-02-24T16:45:54.721 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo nvme connect -t tcp --traddr 10.20.193.151 -s 4420 -n nqn.2016-06.io.spdk:cnode1 2026-02-24T16:45:54.810 INFO:tasks.workunit.client.1.trial052.stdout:connecting to device: nvme3 2026-02-24T16:45:54.814 INFO:tasks.workunit.client.1.trial052.stderr:+ sleep 5 2026-02-24T16:45:54.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:54 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2676615060' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:54 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2676615060' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:55.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:54 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2676615060' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:55.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:55 trial183 ceph-mon[43553]: pgmap v186: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:45:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:55 trial151 ceph-mon[39453]: pgmap v186: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:45:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:55 trial153 ceph-mon[44017]: pgmap v186: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-24T16:45:57.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:57 trial183 ceph-mon[43553]: pgmap v187: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:45:57.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:57 trial151 ceph-mon[39453]: pgmap v187: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:45:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:57 trial153 ceph-mon[44017]: pgmap v187: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 341 B/s wr, 11 op/s 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4066623770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3734300123' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/667448490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4066623770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3734300123' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:58.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/667448490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:59.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4066623770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:59.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3734300123' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:59.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/667448490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:45:59.815 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list --output-format=json 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr:+ output='{ 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "Devices":[ 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme0n1", 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng0n1", 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E2MU200", 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Micron_7450_MTFDKBA960TFR", 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"25174FAD8DBC", 2026-02-24T16:45:59.830 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":34631950336, 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":1875385008, 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":960197124096, 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme1n1", 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng1n1", 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E3MQ000", 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:45:59.831 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"012510B47DC4", 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":157298667520, 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":3750748848, 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme2n1", 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng2n1", 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E3MQ000", 2026-02-24T16:45:59.832 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"012510B47A7F", 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":147882979328, 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":3750748848, 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":12, 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n12", 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n12", 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.833 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":16, 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n16", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n16", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.834 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":20, 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n20", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n20", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.835 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":4, 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n4", 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n4", 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.836 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":8, 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n8", 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n8", 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.837 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr:}' 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '{ 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "Devices":[ 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme0n1", 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng0n1", 2026-02-24T16:45:59.838 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E2MU200", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Micron_7450_MTFDKBA960TFR", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"25174FAD8DBC", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":34631950336, 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":1875385008, 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":960197124096, 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme1n1", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng1n1", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E3MQ000", 2026-02-24T16:45:59.839 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"012510B47DC4", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":157298667520, 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":3750748848, 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":1, 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme2n1", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng2n1", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"E3MQ000", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"MTFDKCC1T9TGP-1BK1JABYY", 2026-02-24T16:45:59.840 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"012510B47A7F", 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":147882979328, 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":3750748848, 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":1920383410176, 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":12, 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n12", 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n12", 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.841 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":16, 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n16", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n16", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.842 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":20, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n20", 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n20", 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.843 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":4, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n4", 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n4", 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.844 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "NameSpace":8, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "DevicePath":"/dev/nvme3n8", 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "GenericPath":"/dev/ng3n8", 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "Firmware":"25.09", 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "ModelNumber":"Ceph bdev Controller", 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "SerialNumber":"Ceph96885192365298", 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "UsedBytes":8589934592, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "MaximumLBA":16777216, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "PhysicalSize":8589934592, 2026-02-24T16:45:59.845 INFO:tasks.workunit.client.1.trial052.stderr: "SectorSize":512 2026-02-24T16:45:59.846 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.846 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.846 INFO:tasks.workunit.client.1.trial052.stderr:}' 2026-02-24T16:45:59.846 INFO:tasks.workunit.client.1.trial052.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:45:59.846 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:45:59.847 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] connect test passed!' 2026-02-24T16:45:59.847 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] connect test passed! 2026-02-24T16:45:59.847 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run list_subsys 1 2026-02-24T16:45:59.848 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:45:59.849 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:45:59.849 INFO:tasks.workunit.client.1.trial052.stderr:+ list_subsys 1 2026-02-24T16:45:59.849 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count=1 2026-02-24T16:45:59.849 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:45:59.849 INFO:tasks.workunit.client.1.trial052.stderr:+ output='[ 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Subsystems":[ 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys0", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme0", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:45:59.850 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:02:00.0", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys1", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme1", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:03:00.0", 2026-02-24T16:45:59.851 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys2", 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F", 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme2", 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:04:00.0", 2026-02-24T16:45:59.852 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys3", 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme3", 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:45:59.853 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:]' 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys3",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"Paths":[' '{' '"Name":"nvme3",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:++ grep -o '"tcp"' 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:+ multipath=1 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 1 -ne 1 ']' 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:45:59.854 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:45:59.855 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:45:59.855 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run disconnect_all 2026-02-24T16:45:59.855 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: disconnect_all 2026-02-24T16:45:59.856 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2026-02-24T16:45:59.856 INFO:tasks.workunit.client.1.trial052.stderr:+ disconnect_all 2026-02-24T16:45:59.856 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo nvme disconnect-all 2026-02-24T16:45:59.964 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:45:59 trial153 ceph-mon[44017]: pgmap v188: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:45:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:45:59 trial183 ceph-mon[43553]: pgmap v188: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:45:59.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:45:59 trial151 ceph-mon[39453]: pgmap v188: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 85 KiB/s rd, 426 B/s wr, 15 op/s 2026-02-24T16:46:00.203 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list 2026-02-24T16:46:00.212 INFO:tasks.workunit.client.1.trial052.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme0n1 /dev/ng0n1 25174FAD8DBC Micron_7450_MTFDKBA960TFR 0x1 34.63 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47DC4 MTFDKCC1T9TGP-1BK1JABYY 0x1 157.30 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47A7F MTFDKCC1T9TGP-1BK1JABYY 0x1 147.88 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme0n1 /dev/ng0n1 25174FAD8DBC Micron_7450_MTFDKBA960TFR 0x1 34.63 GB / 960.20 GB 512 B + 0 B E2MU200 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme1n1 /dev/ng1n1 012510B47DC4 MTFDKCC1T9TGP-1BK1JABYY 0x1 157.30 GB / 1.92 TB 512 B + 0 B E3MQ000 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:/dev/nvme2n1 /dev/ng2n1 012510B47A7F MTFDKCC1T9TGP-1BK1JABYY 0x1 147.88 GB / 1.92 TB 512 B + 0 B E3MQ000 ' 2026-02-24T16:46:00.213 INFO:tasks.workunit.client.1.trial052.stderr:+ grep -q 'Ceph bdev Controller' 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] disconnect_all test passed! 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run list_subsys 0 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ list_subsys 0 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count=0 2026-02-24T16:46:00.214 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr:+ output='[ 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "Subsystems":[ 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys0", 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC", 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.224 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme0", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:02:00.0", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys1", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme1", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:03:00.0", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys2", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme2", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:04:00.0", 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:00.225 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr:]' 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:46:00.226 INFO:tasks.workunit.client.1.trial052.stderr:++ grep -o '"tcp"' 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ multipath=0 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -ne 0 ']' 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ devices_count=60 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run connect_all 60 2026-02-24T16:46:00.227 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: connect_all 2026-02-24T16:46:00.228 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: connect_all' 2026-02-24T16:46:00.228 INFO:tasks.workunit.client.1.trial052.stderr:+ connect_all 60 2026-02-24T16:46:00.228 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo nvme connect-all --traddr=10.20.193.151 --transport=tcp -l 3600 2026-02-24T16:46:00.708 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:00] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:46:00.708 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:00] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:46:00.708 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:00] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:46:00.708 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:00] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:46:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:00] INFO state.py:603 (81): Succeeded to lock OMAP file (shared) after 1 retries 2026-02-24T16:46:00.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2530927637' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/434011787' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1115319852' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3083384833' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:00 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4049438949' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2530927637' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/434011787' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1115319852' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3083384833' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:00.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:00 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4049438949' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2530927637' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/434011787' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1115319852' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3083384833' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:00 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4049438949' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw listeners", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.422 INFO:tasks.workunit.client.1.trial052.stderr:+ sleep 5 2026-02-24T16:46:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:01 trial183 ceph-mon[43553]: pgmap v189: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:46:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/254258692' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:01.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:01 trial151 ceph-mon[39453]: pgmap v189: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:46:01.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/254258692' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:01 trial153 ceph-mon[44017]: pgmap v189: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 426 B/s wr, 11 op/s 2026-02-24T16:46:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/254258692' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:03.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:03 trial183 ceph-mon[43553]: pgmap v190: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 3.0 KiB/s wr, 22 op/s 2026-02-24T16:46:03.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:03 trial151 ceph-mon[39453]: pgmap v190: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 3.0 KiB/s wr, 22 op/s 2026-02-24T16:46:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:03 trial153 ceph-mon[44017]: pgmap v190: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 100 KiB/s rd, 3.0 KiB/s wr, 22 op/s 2026-02-24T16:46:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:05 trial183 ceph-mon[43553]: pgmap v191: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 16 op/s 2026-02-24T16:46:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1067908054' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/502965910' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3480969196' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:05 trial151 ceph-mon[39453]: pgmap v191: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 16 op/s 2026-02-24T16:46:05.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1067908054' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:05.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/502965910' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:05.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3480969196' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:05 trial153 ceph-mon[44017]: pgmap v191: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 3.0 KiB/s wr, 16 op/s 2026-02-24T16:46:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1067908054' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/502965910' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3480969196' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:06.422 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_devices_count=60 2026-02-24T16:46:06.423 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list --output-format=json 2026-02-24T16:46:06.423 INFO:tasks.workunit.client.1.trial052.stderr:++ grep -o 'Ceph bdev Controller' 2026-02-24T16:46:06.424 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:46:06.457 INFO:tasks.workunit.client.1.trial052.stderr:+ actual_devices=60 2026-02-24T16:46:06.458 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 60 -ne 60 ']' 2026-02-24T16:46:06.458 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:46:06.458 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] connect_all test passed!' 2026-02-24T16:46:06.458 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] connect_all test passed! 2026-02-24T16:46:06.459 INFO:tasks.workunit.client.1.trial052.stderr:++ echo 10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:46:06.459 INFO:tasks.workunit.client.1.trial052.stderr:++ tr -cd , 2026-02-24T16:46:06.459 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -c 2026-02-24T16:46:06.459 INFO:tasks.workunit.client.1.trial052.stderr:+ gateways_count=4 2026-02-24T16:46:06.459 INFO:tasks.workunit.client.1.trial052.stderr:+ multipath_count=12 2026-02-24T16:46:06.460 INFO:tasks.workunit.client.1.trial052.stderr:+ test_run list_subsys 12 2026-02-24T16:46:06.460 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] Running test: list_subsys 2026-02-24T16:46:06.460 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] Running test: list_subsys' 2026-02-24T16:46:06.461 INFO:tasks.workunit.client.1.trial052.stderr:+ list_subsys 12 2026-02-24T16:46:06.461 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count=12 2026-02-24T16:46:06.461 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo nvme list-subsys --output-format=json 2026-02-24T16:46:06.480 INFO:tasks.workunit.client.1.trial052.stderr:+ output='[ 2026-02-24T16:46:06.480 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Subsystems":[ 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys0", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme0", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:06.481 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:02:00.0", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys1", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme1", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:03:00.0", 2026-02-24T16:46:06.482 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys8", 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-06.io.spdk:cnode2", 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme10", 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.483 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme11", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme8", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.484 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme9", 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys12", 2026-02-24T16:46:06.485 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-06.io.spdk:cnode3", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme12", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme13", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.486 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme14", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme15", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.487 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys2", 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F", 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme2", 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"pcie", 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"0000:04:00.0", 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.488 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme-subsys4", 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Paths":[ 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme4", 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.489 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme5", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme6", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: }, 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: { 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Name":"nvme7", 2026-02-24T16:46:06.490 INFO:tasks.workunit.client.1.trial052.stderr: "Transport":"tcp", 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: "Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52", 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: "State":"live" 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: ] 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr: } 2026-02-24T16:46:06.491 INFO:tasks.workunit.client.1.trial052.stderr:]' 2026-02-24T16:46:06.492 INFO:tasks.workunit.client.1.trial052.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"HostID":"29b84739-6e95-4147-8ccc-2ecd5be9eabc",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FAD8DBC",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47DC4",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"pcie",' '"Address":"0000:03:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys8",' '"NQN":"nqn.2016-06.io.spdk:cnode2",' '"Paths":[' '{' '"Name":"nvme10",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme11",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme8",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme9",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys12",' '"NQN":"nqn.2016-06.io.spdk:cnode3",' '"Paths":[' '{' '"Name":"nvme12",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme13",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme14",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme15",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B47A7F",' '"Paths":[' '{' '"Name":"nvme2",' '"Transport":"pcie",' '"Address":"0000:04:00.0",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys4",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"Paths":[' '{' '"Name":"nvme4",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.151,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme5",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.153,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme6",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.183,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '},' '{' '"Name":"nvme7",' '"Transport":"tcp",' '"Address":"traddr=10.20.193.191,trsvcid=4420,src_addr=10.20.193.52",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stderr:++ grep -o '"tcp"' 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stderr:+ multipath=12 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 12 -ne 12 ']' 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 0 -eq 0 ']' 2026-02-24T16:46:06.493 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] list_subsys test passed! 2026-02-24T16:46:06.494 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] list_subsys test passed!' 2026-02-24T16:46:06.494 INFO:tasks.workunit.client.1.trial052.stdout:-------------Test Summary------------- 2026-02-24T16:46:06.494 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '-------------Test Summary-------------' 2026-02-24T16:46:06.495 INFO:tasks.workunit.client.1.trial052.stdout:[nvmeof] All nvmeof basic tests passed! 2026-02-24T16:46:06.495 INFO:tasks.workunit.client.1.trial052.stderr:+ echo '[nvmeof] All nvmeof basic tests passed!' 2026-02-24T16:46:06.496 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T16:46:06.496 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2026-02-24T16:46:06.562 INFO:tasks.workunit:Running workunits matching nvmeof/namespace_test.sh on client.1... 2026-02-24T16:46:06.565 INFO:tasks.workunit:Running workunit nvmeof/namespace_test.sh... 2026-02-24T16:46:06.565 DEBUG:teuthology.orchestra.run.trial052:workunit test nvmeof/namespace_test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 IOSTAT_INTERVAL=10 NEW_NAMESPACES_COUNT=5 RBD_POOL=mypool RUNTIME=120 timeout 30m /home/ubuntu/cephtest/clone.client.1/qa/workunits/nvmeof/namespace_test.sh 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:+ source /etc/ceph/nvmeof.env 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_GATEWAY_NAMES=trial151,trial153,trial183,trial191 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=10.20.193.151 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:devel 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2026-02-24T16:46:06.621 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_PORT=4420 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:++ NVMEOF_SRPORT=5500 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:+ RBD_POOL=mypool 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:+ NEW_IMAGE_SIZE=8192 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:+ NEW_NAMESPACES_COUNT=5 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:++ echo 10.20.193.151,10.20.193.153,10.20.193.183,10.20.193.191 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:++ tr -cd , 2026-02-24T16:46:06.622 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -c 2026-02-24T16:46:06.623 INFO:tasks.workunit.client.1.trial052.stderr:+ gateways_count=4 2026-02-24T16:46:06.623 INFO:tasks.workunit.client.1.trial052.stderr:+ new_images_count=15 2026-02-24T16:46:06.623 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 15 2026-02-24T16:46:06.623 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.623 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test1 2026-02-24T16:46:06.624 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test1 --size 8192 2026-02-24T16:46:06.653 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.653 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test2 2026-02-24T16:46:06.653 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test2 --size 8192 2026-02-24T16:46:06.679 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.679 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test3 2026-02-24T16:46:06.679 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test3 --size 8192 2026-02-24T16:46:06.707 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.707 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test4 2026-02-24T16:46:06.707 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test4 --size 8192 2026-02-24T16:46:06.734 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.735 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test5 2026-02-24T16:46:06.735 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test5 --size 8192 2026-02-24T16:46:06.761 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.761 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test6 2026-02-24T16:46:06.761 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test6 --size 8192 2026-02-24T16:46:06.787 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.787 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test7 2026-02-24T16:46:06.787 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test7 --size 8192 2026-02-24T16:46:06.813 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.813 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test8 2026-02-24T16:46:06.813 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test8 --size 8192 2026-02-24T16:46:06.839 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.840 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test9 2026-02-24T16:46:06.840 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test9 --size 8192 2026-02-24T16:46:06.868 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.868 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test10 2026-02-24T16:46:06.868 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test10 --size 8192 2026-02-24T16:46:06.895 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.895 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test11 2026-02-24T16:46:06.895 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test11 --size 8192 2026-02-24T16:46:06.921 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.922 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test12 2026-02-24T16:46:06.922 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test12 --size 8192 2026-02-24T16:46:06.948 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.948 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test13 2026-02-24T16:46:06.948 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test13 --size 8192 2026-02-24T16:46:06.974 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:06.974 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test14 2026-02-24T16:46:06.974 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test14 --size 8192 2026-02-24T16:46:07.002 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $new_images_count) 2026-02-24T16:46:07.002 INFO:tasks.workunit.client.1.trial052.stderr:+ image_name=test15 2026-02-24T16:46:07.002 INFO:tasks.workunit.client.1.trial052.stderr:+ rbd create mypool/test15 --size 8192 2026-02-24T16:46:07.029 INFO:tasks.workunit.client.1.trial052.stderr:+ image_index=1 2026-02-24T16:46:07.029 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 3 2026-02-24T16:46:07.030 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:07.030 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:07.030 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 5 2026-02-24T16:46:07.030 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:07.031 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test1 2026-02-24T16:46:07.031 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image test1 --load-balancing-group 2 2026-02-24T16:46:07.053 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:07.054 INFO:tasks.workunit.client.1.trial052.stderr:Trying to pull quay.io/ceph/nvmeof-cli:devel... 2026-02-24T16:46:07.372 INFO:tasks.workunit.client.1.trial052.stderr:Getting image source signatures 2026-02-24T16:46:07.372 INFO:tasks.workunit.client.1.trial052.stderr:Copying blob sha256:c9bc245d94649254d6eda62be059895c888a58c8efb125021ae7f0fef67431f1 2026-02-24T16:46:07.372 INFO:tasks.workunit.client.1.trial052.stderr:Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2026-02-24T16:46:07.373 INFO:tasks.workunit.client.1.trial052.stderr:Copying blob sha256:c13c0b6a7c5c605d1e299619b7f0853147f52149077394f70a66efdcf16fc6fb 2026-02-24T16:46:07.373 INFO:tasks.workunit.client.1.trial052.stderr:Copying blob sha256:271b82640408a5fd9fe60b060a52f3a26008cf3fd1a7e5ab3e544b00287aa6d4 2026-02-24T16:46:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:07 trial183 ceph-mon[43553]: pgmap v192: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2026-02-24T16:46:07.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:07 trial151 ceph-mon[39453]: pgmap v192: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2026-02-24T16:46:08.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:07 trial153 ceph-mon[44017]: pgmap v192: 33 pgs: 33 active+clean; 609 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 20 op/s 2026-02-24T16:46:08.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:08 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2688082438' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:08.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2688082438' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:09.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:08 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2688082438' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:09.605 INFO:tasks.workunit.client.1.trial052.stderr:Copying config sha256:c788be2c0094e4bcee2bb36f7aaa7cc9bb9c15925ad52cf69bb46c41a4a18334 2026-02-24T16:46:09.605 INFO:tasks.workunit.client.1.trial052.stderr:Writing manifest to image destination 2026-02-24T16:46:09.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:09 trial183 ceph-mon[43553]: pgmap v193: 33 pgs: 33 active+clean; 610 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 88 KiB/s rd, 5.1 KiB/s wr, 27 op/s 2026-02-24T16:46:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:09 trial151 ceph-mon[39453]: pgmap v193: 33 pgs: 33 active+clean; 610 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 88 KiB/s rd, 5.1 KiB/s wr, 27 op/s 2026-02-24T16:46:10.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:09 trial153 ceph-mon[44017]: pgmap v193: 33 pgs: 33 active+clean; 610 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 88 KiB/s rd, 5.1 KiB/s wr, 27 op/s 2026-02-24T16:46:10.278 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 21 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:10.299 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:10.300 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:10.300 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test2 2026-02-24T16:46:10.300 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image test2 --load-balancing-group 3 2026-02-24T16:46:10.338 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:10Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:10.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:36670 2026-02-24T16:46:10.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test1, no need to remove 2026-02-24T16:46:10.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:10.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:10.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 from mypool/test1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:36670 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:10.181656] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO grpc.py:2535 (2): Received request to add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/test1, context: , client address: IPv4 10.20.193.52:36670 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:10.186513] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test1 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:10.186534] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_21, OMAP version: 80 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:10.279945] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:10.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:10.279973] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:10.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:10.983 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO state.py:1684 (81): Start update from 79 to 80 (discovery-trial151). 2026-02-24T16:46:10.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:10 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:10] INFO state.py:2068 (81): Update complete (79 -> 80) (discovery-trial151). 2026-02-24T16:46:10.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO state.py:1684 (2): Start update from 79 to 80 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:10.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 from mypool/test1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO state.py:1684 (105): Start update from 79 to 80 (discovery-trial183). 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO state.py:2068 (105): Update complete (79 -> 80) (discovery-trial183). 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:10.727232] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO grpc.py:2535 (2): Received request to add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test1, context: None 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:10.731988] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test1 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:10.732011] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:10.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:10 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:10] INFO state.py:2068 (2): Update complete (79 -> 80) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO state.py:1684 (132): Start update from 79 to 80 (discovery-trial153). 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO state.py:2068 (132): Update complete (79 -> 80) (discovery-trial153). 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO state.py:1684 (2): Start update from 79 to 80 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 from mypool/test1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:10.726904] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO grpc.py:2535 (2): Received request to add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test1, context: None 2026-02-24T16:46:11.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:10.731812] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test1 2026-02-24T16:46:11.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:10.731838] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:11.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:10 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:10] INFO state.py:2068 (2): Update complete (79 -> 80) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO state.py:1684 (2): Start update from 79 to 80 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 from mypool/test1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO state.py:1684 (126): Start update from 79 to 80 (discovery-trial191). 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO state.py:2068 (126): Update complete (79 -> 80) (discovery-trial191). 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:10.734318] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO grpc.py:2535 (2): Received request to add bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test1, context: None 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:10.739170] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test1 2026-02-24T16:46:11.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:10.739197] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:11.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:10 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:10] INFO state.py:2068 (2): Update complete (79 -> 80) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:11.324 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 22 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:11.345 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:11.345 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:11.345 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test3 2026-02-24T16:46:11.346 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image test3 --load-balancing-group 4 2026-02-24T16:46:11.384 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:11Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:40574 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test2, no need to remove 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 from mypool/test2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:40574 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.228162] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:2535 (2): Received request to add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/test2, context: , client address: IPv4 10.20.193.52:40574 2026-02-24T16:46:11.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.232656] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test2 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.232677] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_22, OMAP version: 81 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.324154] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:11.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.324177] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:11.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.717 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.717 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.978 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-mon[39453]: pgmap v194: 33 pgs: 33 active+clean; 612 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 86 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2026-02-24T16:46:11.978 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:1684 (81): Start update from 80 to 81 (discovery-trial151). 2026-02-24T16:46:11.978 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:2068 (81): Update complete (80 -> 81) (discovery-trial151). 2026-02-24T16:46:11.978 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:60628 2026-02-24T16:46:11.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test3, no need to remove 2026-02-24T16:46:11.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:11.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:11.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b from mypool/test3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:60628 2026-02-24T16:46:11.979 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.981407] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-mon[43553]: pgmap v194: 33 pgs: 33 active+clean; 612 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 86 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO state.py:1684 (2): Start update from 80 to 81 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 from mypool/test2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO state.py:1684 (105): Start update from 80 to 81 (discovery-trial183). 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO state.py:2068 (105): Update complete (80 -> 81) (discovery-trial183). 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:11.771852] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO grpc.py:2535 (2): Received request to add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test2, context: None 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:11.776469] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test2 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:11.776489] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:11.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] INFO state.py:2068 (2): Update complete (80 -> 81) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:11.990 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-mon[44017]: pgmap v194: 33 pgs: 33 active+clean; 612 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 86 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2026-02-24T16:46:11.990 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO state.py:1684 (132): Start update from 80 to 81 (discovery-trial153). 2026-02-24T16:46:11.990 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO state.py:2068 (132): Update complete (80 -> 81) (discovery-trial153). 2026-02-24T16:46:11.990 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO state.py:1684 (2): Start update from 80 to 81 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 from mypool/test2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:11.775926] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO grpc.py:2535 (2): Received request to add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test2, context: None 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:11.780421] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test2 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:11.780442] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] INFO state.py:2068 (2): Update complete (80 -> 81) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:11.991 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:11 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.072 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 23 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO state.py:1684 (2): Start update from 80 to 81 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode1, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 from mypool/test2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO state.py:1684 (126): Start update from 80 to 81 (discovery-trial191). 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO state.py:2068 (126): Update complete (80 -> 81) (discovery-trial191). 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:11.782316] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:12.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO grpc.py:2535 (2): Received request to add bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 3 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test2, context: None 2026-02-24T16:46:12.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:11.800528] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test2 2026-02-24T16:46:12.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:11.800560] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:12.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:11 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:11] INFO state.py:2068 (2): Update complete (80 -> 81) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:12.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.093 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:12.093 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:12.093 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test4 2026-02-24T16:46:12.093 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image test4 --load-balancing-group 1 2026-02-24T16:46:12.135 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:12Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO grpc.py:2535 (2): Received request to add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/test3, context: , client address: IPv4 10.20.193.52:60628 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.986282] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test3 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:11.986305] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_23, OMAP version: 82 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.233 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:11 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.072752] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:12.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.072776] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:12.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:11 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:11] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.778 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1353955615' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.778 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2503480220' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.778 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO state.py:1684 (81): Start update from 81 to 82 (discovery-trial151). 2026-02-24T16:46:12.778 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO state.py:2068 (81): Update complete (81 -> 82) (discovery-trial151). 2026-02-24T16:46:12.778 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:59654 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test4, no need to remove 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 from mypool/test4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:59654 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.690136] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO grpc.py:2535 (2): Received request to add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/test4, context: , client address: IPv4 10.20.193.52:59654 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.694636] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test4 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.694658] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_24, OMAP version: 83 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.779 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.781142] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:12.780 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:12 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:12.781170] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:12.780 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 24 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:12.801 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:12.801 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:12.801 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test5 2026-02-24T16:46:12.801 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image test5 --load-balancing-group 2 2026-02-24T16:46:12.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1353955615' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2503480220' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO state.py:1684 (132): Start update from 81 to 82 (discovery-trial153). 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO state.py:2068 (132): Update complete (81 -> 82) (discovery-trial153). 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO state.py:1684 (2): Start update from 81 to 82 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b from mypool/test3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:12.530850] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:12.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO grpc.py:2535 (2): Received request to add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test3, context: None 2026-02-24T16:46:12.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:12.535806] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test3 2026-02-24T16:46:12.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:12.535827] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:12.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] INFO state.py:2068 (2): Update complete (81 -> 82) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:12.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.838 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:12Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:12.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO state.py:1684 (2): Start update from 81 to 82 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:12.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:12.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b from mypool/test3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:12.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO state.py:1684 (126): Start update from 81 to 82 (discovery-trial191). 2026-02-24T16:46:12.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO state.py:2068 (126): Update complete (81 -> 82) (discovery-trial191). 2026-02-24T16:46:12.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:12.533289] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:12.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO grpc.py:2535 (2): Received request to add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test3, context: None 2026-02-24T16:46:12.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:12.538173] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test3 2026-02-24T16:46:12.843 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:12.538197] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:12.843 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] INFO state.py:2068 (2): Update complete (81 -> 82) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:12.843 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.843 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:12 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:12] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1353955615' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2503480220' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO state.py:1684 (105): Start update from 81 to 82 (discovery-trial183). 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO state.py:2068 (105): Update complete (81 -> 82) (discovery-trial183). 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO state.py:1684 (2): Start update from 81 to 82 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode1, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b from mypool/test3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:12.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:12.530255] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO grpc.py:2535 (2): Received request to add bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b to nqn.2016-06.io.spdk:cnode1 with load balancing group id 4 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test3, context: None 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:12.537412] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test3 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:12.537445] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] INFO state.py:2068 (2): Update complete (81 -> 82) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:12.986 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:12 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:12] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 82 to 83 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 from mypool/test4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:1684 (105): Start update from 82 to 83 (discovery-trial183). 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:2068 (105): Update complete (82 -> 83) (discovery-trial183). 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.234950] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test4, context: None 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.239301] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test4 2026-02-24T16:46:13.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.239322] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:2068 (2): Update complete (82 -> 83) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:1684 (81): Start update from 82 to 83 (discovery-trial151). 2026-02-24T16:46:13.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:2068 (81): Update complete (82 -> 83) (discovery-trial151). 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:58924 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test5, no need to remove 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 from mypool/test5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:58924 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:13.427131] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/test5, context: , client address: IPv4 10.20.193.52:58924 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:13.431650] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test5 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:13.431670] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode1_25, OMAP version: 84 2026-02-24T16:46:13.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.521 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 25 to nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:13.541 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:13.542 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:13.542 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:13.542 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 5 2026-02-24T16:46:13.543 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:13.543 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test6 2026-02-24T16:46:13.543 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image test6 --load-balancing-group 3 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:1684 (132): Start update from 82 to 83 (discovery-trial153). 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:2068 (132): Update complete (82 -> 83) (discovery-trial153). 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 82 to 83 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 from mypool/test4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.236391] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:13.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test4, context: None 2026-02-24T16:46:13.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.240840] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test4 2026-02-24T16:46:13.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.240863] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:13.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:2068 (2): Update complete (82 -> 83) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:13.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.577 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:13Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 82 to 83 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode1, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 from mypool/test4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:1684 (126): Start update from 82 to 83 (discovery-trial191). 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:2068 (126): Update complete (82 -> 83) (discovery-trial191). 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.237997] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f38fa755-e994-4f4a-befe-4412b4fca261 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 1 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test4, context: None 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.242566] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test4 2026-02-24T16:46:13.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.242589] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:13.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:2068 (2): Update complete (82 -> 83) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:13.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:13.945 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:13.521798] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:13.946 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:13.521822] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:13.946 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:1684 (81): Start update from 83 to 84 (discovery-trial151). 2026-02-24T16:46:13.946 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-mon[39453]: pgmap v195: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 143 KiB/s rd, 10 KiB/s wr, 77 op/s 2026-02-24T16:46:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-mon[43553]: pgmap v195: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 143 KiB/s rd, 10 KiB/s wr, 77 op/s 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:1684 (105): Start update from 83 to 84 (discovery-trial183). 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:2068 (105): Update complete (83 -> 84) (discovery-trial183). 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 83 to 84 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 from mypool/test5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:13.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.971105] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:13.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test5, context: None 2026-02-24T16:46:13.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.975478] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test5 2026-02-24T16:46:13.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:13 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:13.975500] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:14.025 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-mon[44017]: pgmap v195: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 143 KiB/s rd, 10 KiB/s wr, 77 op/s 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:1684 (132): Start update from 83 to 84 (discovery-trial153). 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:2068 (132): Update complete (83 -> 84) (discovery-trial153). 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 83 to 84 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 from mypool/test5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.973275] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:14.026 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test5, context: None 2026-02-24T16:46:14.027 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.977738] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test5 2026-02-24T16:46:14.027 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:13 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:13.977763] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:14.027 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (83 -> 84) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:14.198 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 21 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:14.217 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:14.217 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:14.217 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test7 2026-02-24T16:46:14.217 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image test7 --load-balancing-group 4 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:13 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:13] INFO state.py:2068 (81): Update complete (83 -> 84) (discovery-trial151). 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:54248 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test6, no need to remove 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff from mypool/test6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:54248 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.103296] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:2535 (2): Received request to add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/test6, context: , client address: IPv4 10.20.193.52:54248 2026-02-24T16:46:14.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.107692] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test6 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.107715] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_21, OMAP version: 85 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.198927] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:14.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.198955] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:14.259 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:14.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:1684 (126): Start update from 83 to 84 (discovery-trial191). 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:2068 (126): Update complete (83 -> 84) (discovery-trial191). 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO state.py:1684 (2): Start update from 83 to 84 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode1, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 from mypool/test5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.978641] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:13] INFO grpc.py:2535 (2): Received request to add bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test5, context: None 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.982988] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test5 2026-02-24T16:46:14.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:13 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:13.983009] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:14.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (83 -> 84) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:14.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (83 -> 84) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.940 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:1684 (81): Start update from 84 to 85 (discovery-trial151). 2026-02-24T16:46:14.940 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:2068 (81): Update complete (84 -> 85) (discovery-trial151). 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:50508 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test7, no need to remove 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 from mypool/test7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:50508 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.840058] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO grpc.py:2535 (2): Received request to add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/test7, context: , client address: IPv4 10.20.193.52:50508 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.844726] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test7 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.844748] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_22, OMAP version: 86 2026-02-24T16:46:14.941 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:14] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.943549] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:14.942 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:14 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:14.943576] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:14.942 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 22 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:14.964 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:14.964 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:14.964 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test8 2026-02-24T16:46:14.964 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image test8 --load-balancing-group 1 2026-02-24T16:46:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO state.py:1684 (105): Start update from 84 to 85 (discovery-trial183). 2026-02-24T16:46:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO state.py:2068 (105): Update complete (84 -> 85) (discovery-trial183). 2026-02-24T16:46:14.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO state.py:1684 (2): Start update from 84 to 85 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff from mypool/test6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:14.651886] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO grpc.py:2535 (2): Received request to add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test6, context: None 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:14.656482] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test6 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:14.656504] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (84 -> 85) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:14 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:14.999 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:15Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO state.py:1684 (132): Start update from 84 to 85 (discovery-trial153). 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO state.py:2068 (132): Update complete (84 -> 85) (discovery-trial153). 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO state.py:1684 (2): Start update from 84 to 85 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff from mypool/test6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:14.654423] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO grpc.py:2535 (2): Received request to add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test6, context: None 2026-02-24T16:46:15.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:14.675593] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test6 2026-02-24T16:46:15.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:14.675623] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:15.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (84 -> 85) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:15.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:14 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:14] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO state.py:1684 (2): Start update from 84 to 85 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff from mypool/test6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO state.py:1684 (126): Start update from 84 to 85 (discovery-trial191). 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO state.py:2068 (126): Update complete (84 -> 85) (discovery-trial191). 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:14.654880] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO grpc.py:2535 (2): Received request to add bdev_7f04c0eb-d018-4904-938d-15e96a4242ff to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test6, context: None 2026-02-24T16:46:15.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:14.659306] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test6 2026-02-24T16:46:15.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:14.659329] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:15.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] INFO state.py:2068 (2): Update complete (84 -> 85) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:15.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:14 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:14] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO state.py:1684 (81): Start update from 85 to 86 (discovery-trial151). 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO state.py:2068 (81): Update complete (85 -> 86) (discovery-trial151). 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:36290 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test8, no need to remove 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:15.645 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 from mypool/test8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:36290 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:15.560445] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO grpc.py:2535 (2): Received request to add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/test8, context: , client address: IPv4 10.20.193.52:36290 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:15.564920] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test8 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:15.564940] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_23, OMAP version: 87 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:15.648064] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:15.646 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:15.648090] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:15.647 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 23 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:15.666 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:15.666 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:15.666 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test9 2026-02-24T16:46:15.666 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image test9 --load-balancing-group 2 2026-02-24T16:46:15.708 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:15Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:15.730 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO state.py:1684 (105): Start update from 85 to 86 (discovery-trial183). 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO state.py:2068 (105): Update complete (85 -> 86) (discovery-trial183). 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO state.py:1684 (2): Start update from 85 to 86 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 from mypool/test7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:15.385000] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO grpc.py:2535 (2): Received request to add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test7, context: None 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:15.389424] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test7 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:15.389444] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:15.731 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] INFO state.py:2068 (2): Update complete (85 -> 86) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:15] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO state.py:1684 (2): Start update from 85 to 86 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 from mypool/test7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:15.732 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO state.py:1684 (132): Start update from 85 to 86 (discovery-trial153). 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO state.py:2068 (132): Update complete (85 -> 86) (discovery-trial153). 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:15.383742] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO grpc.py:2535 (2): Received request to add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test7, context: None 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:15.388237] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test7 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:15.388260] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] INFO state.py:2068 (2): Update complete (85 -> 86) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.733 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO state.py:1684 (126): Start update from 85 to 86 (discovery-trial191). 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO state.py:2068 (126): Update complete (85 -> 86) (discovery-trial191). 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO state.py:1684 (2): Start update from 85 to 86 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode2, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 from mypool/test7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:15.391930] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:15.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO grpc.py:2535 (2): Received request to add bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 4 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test7, context: None 2026-02-24T16:46:15.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:15.396306] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test7 2026-02-24T16:46:15.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:15.396328] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:15.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] INFO state.py:2068 (2): Update complete (85 -> 86) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:15.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:15 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:15] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-mon[43553]: pgmap v196: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 105 KiB/s rd, 7.6 KiB/s wr, 65 op/s 2026-02-24T16:46:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2979553058' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-mon[39453]: pgmap v196: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 105 KiB/s rd, 7.6 KiB/s wr, 65 op/s 2026-02-24T16:46:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2979553058' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:16.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-mon[44017]: pgmap v196: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 105 KiB/s rd, 7.6 KiB/s wr, 65 op/s 2026-02-24T16:46:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2979553058' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:1684 (126): Start update from 86 to 87 (discovery-trial191). 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:2068 (126): Update complete (86 -> 87) (discovery-trial191). 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 86 to 87 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 from mypool/test8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.112259] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test8, context: None 2026-02-24T16:46:16.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.116699] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test8 2026-02-24T16:46:16.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.116722] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:16.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (86 -> 87) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:16.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.359 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:1684 (81): Start update from 86 to 87 (discovery-trial151). 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:2068 (81): Update complete (86 -> 87) (discovery-trial151). 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:41738 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test9, no need to remove 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 from mypool/test9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:41738 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.272895] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40d6f514-4245-4aa7-b64b-eef850664928 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_40d6f514-4245-4aa7-b64b-eef850664928 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/test9, context: , client address: IPv4 10.20.193.52:41738 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.277389] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test9 2026-02-24T16:46:16.360 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.277410] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:16.361 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_24, OMAP version: 88 2026-02-24T16:46:16.361 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.361 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.361 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.362964] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:16.361 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.362988] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:16.361 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 24 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:16.382 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:16.382 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:16.383 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test10 2026-02-24T16:46:16.383 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image test10 --load-balancing-group 3 2026-02-24T16:46:16.422 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:16Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:16.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 86 to 87 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 from mypool/test8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:1684 (105): Start update from 86 to 87 (discovery-trial183). 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:2068 (105): Update complete (86 -> 87) (discovery-trial183). 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.105272] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test8, context: None 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.109747] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test8 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.109769] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (86 -> 87) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:1684 (132): Start update from 86 to 87 (discovery-trial153). 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:2068 (132): Update complete (86 -> 87) (discovery-trial153). 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 86 to 87 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode2, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 from mypool/test8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.108479] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 1 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test8, context: None 2026-02-24T16:46:16.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.112984] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test8 2026-02-24T16:46:16.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.113009] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:16.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (86 -> 87) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:16.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2348389887' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:1684 (105): Start update from 87 to 88 (discovery-trial183). 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:2068 (105): Update complete (87 -> 88) (discovery-trial183). 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 87 to 88 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 from mypool/test9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.817956] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40d6f514-4245-4aa7-b64b-eef850664928 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_40d6f514-4245-4aa7-b64b-eef850664928 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test9, context: None 2026-02-24T16:46:16.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.822436] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test9 2026-02-24T16:46:16.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:16.822456] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:16.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (87 -> 88) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:16.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2348389887' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:1684 (81): Start update from 87 to 88 (discovery-trial151). 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:2068 (81): Update complete (87 -> 88) (discovery-trial151). 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:56848 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test10, no need to remove 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 from mypool/test10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:56848 2026-02-24T16:46:16.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.978868] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/test10, context: , client address: IPv4 10.20.193.52:56848 2026-02-24T16:46:16.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.983373] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test10 2026-02-24T16:46:16.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:16.983393] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:16.987 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2348389887' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:1684 (132): Start update from 87 to 88 (discovery-trial153). 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:2068 (132): Update complete (87 -> 88) (discovery-trial153). 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 87 to 88 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 from mypool/test9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.816798] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40d6f514-4245-4aa7-b64b-eef850664928 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_40d6f514-4245-4aa7-b64b-eef850664928 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test9, context: None 2026-02-24T16:46:16.988 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.821382] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test9 2026-02-24T16:46:16.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:16.821407] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:16.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (87 -> 88) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:16.989 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.075 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 25 to nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:1684 (126): Start update from 87 to 88 (discovery-trial191). 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:2068 (126): Update complete (87 -> 88) (discovery-trial191). 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:1684 (2): Start update from 87 to 88 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode2, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 from mypool/test9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.824403] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_40d6f514-4245-4aa7-b64b-eef850664928 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO grpc.py:2535 (2): Received request to add bdev_40d6f514-4245-4aa7-b64b-eef850664928 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 2 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test9, context: None 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.828891] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test9 2026-02-24T16:46:17.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:16.828913] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:17.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] INFO state.py:2068 (2): Update complete (87 -> 88) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:17.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:16 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:16] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.095 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:17.095 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:17.095 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:17.096 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 5 2026-02-24T16:46:17.096 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:17.097 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test11 2026-02-24T16:46:17.097 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image test11 --load-balancing-group 4 2026-02-24T16:46:17.130 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:17Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:17.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:16 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:16] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:16 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode2_25, OMAP version: 89 2026-02-24T16:46:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:16 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:16] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.075443] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:17.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.075474] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:17.768 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO state.py:1684 (81): Start update from 88 to 89 (discovery-trial151). 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO state.py:2068 (81): Update complete (88 -> 89) (discovery-trial151). 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:53482 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test11, no need to remove 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf from mypool/test11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:53482 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.716241] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO grpc.py:2535 (2): Received request to add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/test11, context: , client address: IPv4 10.20.193.52:53482 2026-02-24T16:46:17.769 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.720684] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test11 2026-02-24T16:46:17.770 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.720704] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:17.770 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_21, OMAP version: 90 2026-02-24T16:46:17.770 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.770 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:17] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.770 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO state.py:1684 (2): Start update from 88 to 89 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 from mypool/test10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO state.py:1684 (132): Start update from 88 to 89 (discovery-trial153). 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO state.py:2068 (132): Update complete (88 -> 89) (discovery-trial153). 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:17.522422] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO grpc.py:2535 (2): Received request to add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test10, context: None 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:17.528604] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test10 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:17.528628] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] INFO state.py:2068 (2): Update complete (88 -> 89) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:17.771 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:17] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO state.py:1684 (105): Start update from 88 to 89 (discovery-trial183). 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO state.py:2068 (105): Update complete (88 -> 89) (discovery-trial183). 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO state.py:1684 (2): Start update from 88 to 89 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 from mypool/test10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:17.526338] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO grpc.py:2535 (2): Received request to add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test10, context: None 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:17.530840] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test10 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:17.530862] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:17.772 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] INFO state.py:2068 (2): Update complete (88 -> 89) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:17.773 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.773 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:17] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.817 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 21 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:17.836 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:17.837 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:17.837 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test12 2026-02-24T16:46:17.837 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image test12 --load-balancing-group 1 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO state.py:1684 (2): Start update from 88 to 89 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode2, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 from mypool/test10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO state.py:1684 (126): Start update from 88 to 89 (discovery-trial191). 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO state.py:2068 (126): Update complete (88 -> 89) (discovery-trial191). 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:17.526303] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO grpc.py:2535 (2): Received request to add bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 to nqn.2016-06.io.spdk:cnode2 with load balancing group id 3 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test10, context: None 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:17.545515] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test10 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:17.545551] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:17.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] INFO state.py:2068 (2): Update complete (88 -> 89) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:17.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:17 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:17] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:17.873 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:17Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:18.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:17 trial153 ceph-mon[44017]: pgmap v197: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 390 KiB/s rd, 8.0 KiB/s wr, 415 op/s 2026-02-24T16:46:18.232 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:17 trial183 ceph-mon[43553]: pgmap v197: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 390 KiB/s rd, 8.0 KiB/s wr, 415 op/s 2026-02-24T16:46:18.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-mon[39453]: pgmap v197: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 390 KiB/s rd, 8.0 KiB/s wr, 415 op/s 2026-02-24T16:46:18.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.818175] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:18.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:17 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:17.818202] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:18.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:1684 (105): Start update from 89 to 90 (discovery-trial183). 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:2068 (105): Update complete (89 -> 90) (discovery-trial183). 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 89 to 90 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf from mypool/test11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:18.263242] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO grpc.py:2535 (2): Received request to add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test11, context: None 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:18.271373] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test11 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:18.271399] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:18.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:2068 (2): Update complete (89 -> 90) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:18.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:1684 (81): Start update from 89 to 90 (discovery-trial151). 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:2068 (81): Update complete (89 -> 90) (discovery-trial151). 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:57672 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test12, no need to remove 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_29461d02-252c-42e1-8f87-67b48090adee from mypool/test12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:57672 2026-02-24T16:46:18.547 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:18.450625] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_29461d02-252c-42e1-8f87-67b48090adee rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO grpc.py:2535 (2): Received request to add bdev_29461d02-252c-42e1-8f87-67b48090adee to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1, auto_visible: True, auto_resize: True, RBD image mypool/test12, context: , client address: IPv4 10.20.193.52:57672 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:18.455825] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test12 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:18.455857] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_22, OMAP version: 91 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:18.550083] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:18.548 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:18.550114] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:18.549 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 22 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:1684 (132): Start update from 89 to 90 (discovery-trial153). 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:2068 (132): Update complete (89 -> 90) (discovery-trial153). 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 89 to 90 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf from mypool/test11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:18.259249] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO grpc.py:2535 (2): Received request to add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test11, context: None 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:18.263790] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test11 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:18.263814] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:2068 (2): Update complete (89 -> 90) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:18.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.570 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:18.571 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:18.571 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test13 2026-02-24T16:46:18.571 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image test13 --load-balancing-group 2 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:1684 (126): Start update from 89 to 90 (discovery-trial191). 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:2068 (126): Update complete (89 -> 90) (discovery-trial191). 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 89 to 90 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 21 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf from mypool/test11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:18.269337] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO grpc.py:2535 (2): Received request to add bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 21, auto_visible: True, auto_resize: True, RBD image mypool/test11, context: None 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:18.273899] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test11 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:18.273920] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 21 2026-02-24T16:46:18.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:2068 (2): Update complete (89 -> 90) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:18.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:18.610 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:1684 (81): Start update from 90 to 91 (discovery-trial151). 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:18 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:18] INFO state.py:2068 (81): Update complete (90 -> 91) (discovery-trial151). 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:49438 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test13, no need to remove 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:19.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 from mypool/test13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:49438 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.187307] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2, auto_visible: True, auto_resize: True, RBD image mypool/test13, context: , client address: IPv4 10.20.193.52:49438 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.191735] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test13 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.191755] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_23, OMAP version: 92 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 90 to 91 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:19.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_29461d02-252c-42e1-8f87-67b48090adee from mypool/test12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:1684 (105): Start update from 90 to 91 (discovery-trial183). 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:18] INFO state.py:2068 (105): Update complete (90 -> 91) (discovery-trial183). 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.010247] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_29461d02-252c-42e1-8f87-67b48090adee rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_29461d02-252c-42e1-8f87-67b48090adee to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test12, context: None 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.021737] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test12 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.021762] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (90 -> 91) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.236 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.284 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 23 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:19.304 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:19.304 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:19.305 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test14 2026-02-24T16:46:19.305 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image test14 --load-balancing-group 3 2026-02-24T16:46:19.313 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:1684 (132): Start update from 90 to 91 (discovery-trial153). 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:2068 (132): Update complete (90 -> 91) (discovery-trial153). 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 90 to 91 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:18 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_29461d02-252c-42e1-8f87-67b48090adee from mypool/test12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.000278] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_29461d02-252c-42e1-8f87-67b48090adee rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_29461d02-252c-42e1-8f87-67b48090adee to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test12, context: None 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.004822] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test12 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.004847] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (90 -> 91) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:19.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:1684 (126): Start update from 90 to 91 (discovery-trial191). 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:2068 (126): Update complete (90 -> 91) (discovery-trial191). 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO state.py:1684 (2): Start update from 90 to 91 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO grpc.py:2774 (2): Received request to add namespace 22 to nqn.2016-06.io.spdk:cnode3, ana group 1, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:18 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:18] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_29461d02-252c-42e1-8f87-67b48090adee from mypool/test12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.001715] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_29461d02-252c-42e1-8f87-67b48090adee rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_29461d02-252c-42e1-8f87-67b48090adee to nqn.2016-06.io.spdk:cnode3 with load balancing group id 1 using ID 22, auto_visible: True, auto_resize: True, RBD image mypool/test12, context: None 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.006267] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test12 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.006294] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 22 2026-02-24T16:46:19.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (90 -> 91) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:19.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.346 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:19.707 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.285813] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:19.707 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.285839] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:19.707 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:1684 (81): Start update from 91 to 92 (discovery-trial151). 2026-02-24T16:46:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-mon[43553]: pgmap v198: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 7.3 KiB/s wr, 571 op/s 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO state.py:1684 (105): Start update from 91 to 92 (discovery-trial183). 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO state.py:2068 (105): Update complete (91 -> 92) (discovery-trial183). 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO state.py:1684 (2): Start update from 91 to 92 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 from mypool/test13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.744266] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test13, context: None 2026-02-24T16:46:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.748734] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test13 2026-02-24T16:46:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:19.748755] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (91 -> 92) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:19 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.986 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-mon[39453]: pgmap v198: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 7.3 KiB/s wr, 571 op/s 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:2068 (81): Update complete (91 -> 92) (discovery-trial151). 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:54714 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test14, no need to remove 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 from mypool/test14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:54714 2026-02-24T16:46:19.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.903636] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3, auto_visible: True, auto_resize: True, RBD image mypool/test14, context: , client address: IPv4 10.20.193.52:54714 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.917802] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test14 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:19.917831] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_24, OMAP version: 93 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:19.987 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:19 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.012 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 24 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:20.033 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:20.033 INFO:tasks.workunit.client.1.trial052.stderr:+ for ns in $(seq 1 $NEW_NAMESPACES_COUNT) 2026-02-24T16:46:20.034 INFO:tasks.workunit.client.1.trial052.stderr:+ image=test15 2026-02-24T16:46:20.034 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image test15 --load-balancing-group 4 2026-02-24T16:46:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-mon[44017]: pgmap v198: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 681 KiB/s rd, 7.3 KiB/s wr, 571 op/s 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO state.py:1684 (132): Start update from 91 to 92 (discovery-trial153). 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO state.py:2068 (132): Update complete (91 -> 92) (discovery-trial153). 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO state.py:1684 (2): Start update from 91 to 92 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 from mypool/test13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.732695] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test13, context: None 2026-02-24T16:46:20.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.737254] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test13 2026-02-24T16:46:20.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:19.737278] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:20.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (91 -> 92) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:20.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:19 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.070 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:20Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO state.py:1684 (126): Start update from 91 to 92 (discovery-trial191). 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO state.py:2068 (126): Update complete (91 -> 92) (discovery-trial191). 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO state.py:1684 (2): Start update from 91 to 92 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO grpc.py:2774 (2): Received request to add namespace 23 to nqn.2016-06.io.spdk:cnode3, ana group 2, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 from mypool/test13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.738180] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO grpc.py:2535 (2): Received request to add bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 2 using ID 23, auto_visible: True, auto_resize: True, RBD image mypool/test13, context: None 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.742518] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test13 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:19.742543] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 23 2026-02-24T16:46:20.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] INFO state.py:2068 (2): Update complete (91 -> 92) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:20.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:19 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:19] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.429 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.012396] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:20.429 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.012422] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:20.429 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO state.py:1684 (81): Start update from 92 to 93 (discovery-trial151). 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO state.py:2068 (81): Update complete (92 -> 93) (discovery-trial151). 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO grpc.py:2774 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: , client address: IPv4 10.20.193.52:36622 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO cephutils.py:599 (2): No metadata reservation_key for image mypool/test15, no need to remove 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO cephutils.py:122 (2): Caching response of the monitor: [1, 2, 3, 4] 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:20.693 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 from mypool/test15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 10.20.193.52:36622 2026-02-24T16:46:20.694 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.696389] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO state.py:1684 (105): Start update from 92 to 93 (discovery-trial183). 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO state.py:2068 (105): Update complete (92 -> 93) (discovery-trial183). 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO state.py:1684 (2): Start update from 92 to 93 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 from mypool/test14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:20.704 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:20.460331] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO grpc.py:2535 (2): Received request to add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test14, context: None 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:20.464744] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test14 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:20.464765] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] INFO state.py:2068 (2): Update complete (92 -> 93) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.705 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO state.py:1684 (126): Start update from 92 to 93 (discovery-trial191). 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO state.py:2068 (126): Update complete (92 -> 93) (discovery-trial191). 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO state.py:1684 (2): Start update from 92 to 93 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 from mypool/test14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:20.462904] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO grpc.py:2535 (2): Received request to add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test14, context: None 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:20.467398] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test14 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:20.467422] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] INFO state.py:2068 (2): Update complete (92 -> 93) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:20.706 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO state.py:1684 (2): Start update from 92 to 93 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO state.py:1684 (132): Start update from 92 to 93 (discovery-trial153). 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO state.py:2068 (132): Update complete (92 -> 93) (discovery-trial153). 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO grpc.py:2774 (2): Received request to add namespace 24 to nqn.2016-06.io.spdk:cnode3, ana group 3, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 from mypool/test14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:20.461452] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:20.707 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO grpc.py:2535 (2): Received request to add bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 3 using ID 24, auto_visible: True, auto_resize: True, RBD image mypool/test14, context: None 2026-02-24T16:46:20.708 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:20.465883] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test14 2026-02-24T16:46:20.708 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:20.465908] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 24 2026-02-24T16:46:20.708 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] INFO state.py:2068 (2): Update complete (92 -> 93) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:20.708 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.793 INFO:tasks.workunit.client.1.trial052.stdout:Adding namespace 25 to nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:20.814 INFO:tasks.workunit.client.1.trial052.stderr:+ (( image_index++ )) 2026-02-24T16:46:20.814 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 3 2026-02-24T16:46:20.815 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:20.815 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:20.816 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:20.855 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:20Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:20.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3158935652' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:20.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:20 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3158935652' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO grpc.py:2535 (2): Received request to add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4, auto_visible: True, auto_resize: True, RBD image mypool/test15, context: , client address: IPv4 10.20.193.52:36622 2026-02-24T16:46:20.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.700847] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test15 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.700869] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] INFO state.py:1053 (2): omap_key generated: namespace_nqn.2016-06.io.spdk:cnode3_25, OMAP version: 94 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.794435] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:20.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:20 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:20.794468] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:21.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3158935652' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:21.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:20 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:21.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:20 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:20] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO state.py:1684 (105): Start update from 93 to 94 (discovery-trial183). 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO state.py:2068 (105): Update complete (93 -> 94) (discovery-trial183). 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO state.py:1684 (2): Start update from 93 to 94 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 from mypool/test15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:21.244702] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO grpc.py:2535 (2): Received request to add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test15, context: None 2026-02-24T16:46:21.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:21.249239] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test15 2026-02-24T16:46:21.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:21.249260] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:21.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:21] INFO state.py:2068 (2): Update complete (93 -> 94) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:21.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:21] INFO state.py:1684 (81): Start update from 93 to 94 (discovery-trial151). 2026-02-24T16:46:21.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:21] INFO state.py:2068 (81): Update complete (93 -> 94) (discovery-trial151). 2026-02-24T16:46:21.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:21 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:21] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:58058 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO state.py:1684 (132): Start update from 93 to 94 (discovery-trial153). 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO state.py:2068 (132): Update complete (93 -> 94) (discovery-trial153). 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO state.py:1684 (2): Start update from 93 to 94 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 from mypool/test15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:21.241837] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO grpc.py:2535 (2): Received request to add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test15, context: None 2026-02-24T16:46:21.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:21.246296] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test15 2026-02-24T16:46:21.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:21.246322] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:21.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:21] INFO state.py:2068 (2): Update complete (93 -> 94) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO state.py:1684 (126): Start update from 93 to 94 (discovery-trial191). 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO state.py:2068 (126): Update complete (93 -> 94) (discovery-trial191). 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO state.py:1684 (2): Start update from 93 to 94 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO grpc.py:2774 (2): Received request to add namespace 25 to nqn.2016-06.io.spdk:cnode3, ana group 4, no_auto_visible: False, disable_auto_resize: False, read_only: False, location: "", encryption_entries: [], encryption_algorithm: no_algorithm, context: None 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO grpc.py:1509 (2): Received request to create read write bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 from mypool/test15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:21.251547] bdev_rbd.c:1757:bdev_rbd_create: *NOTICE*: Add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 rbd disk to lun , cluster-id 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO grpc.py:2535 (2): Received request to add bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 to nqn.2016-06.io.spdk:cnode3 with load balancing group id 4 using ID 25, auto_visible: True, auto_resize: True, RBD image mypool/test15, context: None 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:21.256001] bdev_rbd.c:1935:bdev_rbd_ns_reservation_load_json: *NOTICE*: Failed to get metadata key = reservation_key rbd-name test15 2026-02-24T16:46:21.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:21.256023] subsystem_rbd_resrv.c: 160:ns_rdb_load: *NOTICE*: Subsystem load reservation failed, rc -2, ns 25 2026-02-24T16:46:21.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:21 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:21] INFO state.py:2068 (2): Update complete (93 -> 94) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode1: 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout:NQN NSID Bdev RBD Mode Image Block UUID Load Location Visibility Encryption IOs per R/W, R, W MBs 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout: Name Image Size Size Balancing second per second 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout: Group 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 1 bdev_840516f4-691c- mypool/myimage1 Read-Write 8 GiB 512 Bytes 840516f4-691c-4ba5- 2 All Hosts None unset unset 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout: 4ba5-a1ba-efa5fa20e5a1 a1ba-efa5fa20e5a1 unset 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.933 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 2 bdev_cfb31ce7-cb91- mypool/myimage2 Read-Write 8 GiB 512 Bytes cfb31ce7-cb91-4b97- 3 All Hosts None unset unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: 4b97-adf8-8272506ab870 adf8-8272506ab870 unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 3 bdev_40fce57c-8276- mypool/myimage3 Read-Write 8 GiB 512 Bytes 40fce57c-8276-4a72- 4 All Hosts None unset unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: 4a72-b45a-7be331c8e74f b45a-7be331c8e74f unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 4 bdev_0d16b116-783f- mypool/myimage4 Read-Write 8 GiB 512 Bytes 0d16b116-783f-43cc- 1 All Hosts None unset unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: 43cc-a1cb-37afe0fd7c76 a1cb-37afe0fd7c76 unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 5 bdev_b3b53051-701d- mypool/myimage5 Read-Write 8 GiB 512 Bytes b3b53051-701d-48ff- 2 All Hosts None unset unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: 48ff-aa3e-45a9ad76db11 aa3e-45a9ad76db11 unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.934 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 6 bdev_e988432a-1296- mypool/myimage6 Read-Write 8 GiB 512 Bytes e988432a-1296-42f2- 3 All Hosts None unset unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: 42f2-b6da-fcaadbb7d5f9 b6da-fcaadbb7d5f9 unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 7 bdev_fbfcd850-798d- mypool/myimage7 Read-Write 8 GiB 512 Bytes fbfcd850-798d-4a37- 4 All Hosts None unset unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: 4a37-bc95-da6914211a74 bc95-da6914211a74 unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 8 bdev_1487f5af-dbee- mypool/myimage8 Read-Write 8 GiB 512 Bytes 1487f5af-dbee-446c- 1 All Hosts None unset unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: 446c-8277-9e3e8cc61d9a 8277-9e3e8cc61d9a unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 9 bdev_ccdaa0e7-2dd6- mypool/myimage9 Read-Write 8 GiB 512 Bytes ccdaa0e7-2dd6-4d1c- 2 All Hosts None unset unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: 4d1c-bc1d-3a9533c7f552 bc1d-3a9533c7f552 unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.935 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 10 bdev_967ce80d-eb27- mypool/myimage10 Read-Write 8 GiB 512 Bytes 967ce80d-eb27-4704- 3 All Hosts None unset unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: 4704-9970-b9c49e6ba94e 9970-b9c49e6ba94e unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 11 bdev_b1ff6657-0bd2- mypool/myimage11 Read-Write 8 GiB 512 Bytes b1ff6657-0bd2-4a14- 4 All Hosts None unset unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: 4a14-a224-d9518b9e0e8b a224-d9518b9e0e8b unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 12 bdev_fcc74cc6-f263- mypool/myimage12 Read-Write 8 GiB 512 Bytes fcc74cc6-f263-426a- 1 All Hosts None unset unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: 426a-acff-c8c762707e26 acff-c8c762707e26 unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 13 bdev_2d6ea4fa-a156- mypool/myimage13 Read-Write 8 GiB 512 Bytes 2d6ea4fa-a156-4fb6- 2 All Hosts None unset unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: 4fb6-b260-22c7c4ea52d6 b260-22c7c4ea52d6 unset 2026-02-24T16:46:21.936 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 14 bdev_a8513d36-6a5a- mypool/myimage14 Read-Write 8 GiB 512 Bytes a8513d36-6a5a-4fb4- 3 All Hosts None unset unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: 4fb4-86c4-a114c780b45e 86c4-a114c780b45e unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 15 bdev_047869d6-7b93- mypool/myimage15 Read-Write 8 GiB 512 Bytes 047869d6-7b93-44d0- 4 All Hosts None unset unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: 44d0-8cef-79878331cc42 8cef-79878331cc42 unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 16 bdev_d68a8f73-7e96- mypool/myimage16 Read-Write 8 GiB 512 Bytes d68a8f73-7e96-4889- 1 All Hosts None unset unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: 4889-b293-122ed5581ea3 b293-122ed5581ea3 unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 17 bdev_d9d070fa-74fd- mypool/myimage17 Read-Write 8 GiB 512 Bytes d9d070fa-74fd-4ec2- 2 All Hosts None unset unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: 4ec2-a7eb-bd8b8d4b2c5d a7eb-bd8b8d4b2c5d unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.937 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 18 bdev_c883aa1e-10be- mypool/myimage18 Read-Write 8 GiB 512 Bytes c883aa1e-10be-4039- 3 All Hosts None unset unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: 4039-b95e-fc44869c1f47 b95e-fc44869c1f47 unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 19 bdev_5980b9b8-796e- mypool/myimage19 Read-Write 8 GiB 512 Bytes 5980b9b8-796e-49f0- 4 All Hosts None unset unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: 49f0-a65d-cd0f3161347f a65d-cd0f3161347f unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 20 bdev_e3fd8e09-33ec- mypool/myimage20 Read-Write 8 GiB 512 Bytes e3fd8e09-33ec-4b92- 1 All Hosts None unset unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: 4b92-9142-92798b8a4431 9142-92798b8a4431 unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 21 bdev_83e00b6e-24fd- mypool/test1 Read-Write 8 GiB 512 Bytes 83e00b6e-24fd-4352- 2 All Hosts None unset unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: 4352-8e44-3a7f06bf26c5 8e44-3a7f06bf26c5 unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 22 bdev_b6118346-75e6- mypool/test2 Read-Write 8 GiB 512 Bytes b6118346-75e6-43bd- 3 All Hosts None unset unset 2026-02-24T16:46:21.938 INFO:tasks.workunit.client.1.trial052.stdout: 43bd-bf7c-1f36e66325d6 bf7c-1f36e66325d6 unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 23 bdev_7491df4d-8f09- mypool/test3 Read-Write 8 GiB 512 Bytes 7491df4d-8f09-4e64- 4 All Hosts None unset unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: 4e64-a6b1-9e16d03ee72b a6b1-9e16d03ee72b unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 24 bdev_f38fa755-e994- mypool/test4 Read-Write 8 GiB 512 Bytes f38fa755-e994-4f4a- 1 All Hosts None unset unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: 4f4a-befe-4412b4fca261 befe-4412b4fca261 unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode1 25 bdev_f4382a38-b1e9- mypool/test5 Read-Write 8 GiB 512 Bytes f4382a38-b1e9-4be8- 2 All Hosts None unset unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: 4be8-8684-47dd99dff410 8684-47dd99dff410 unset 2026-02-24T16:46:21.939 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:21.954 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:21.954 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:21.955 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:21.994 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:21Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-mon[44017]: pgmap v199: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 755 KiB/s rd, 7.1 KiB/s wr, 685 op/s 2026-02-24T16:46:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:21 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2797722501' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:22.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-mon[43553]: pgmap v199: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 755 KiB/s rd, 7.1 KiB/s wr, 685 op/s 2026-02-24T16:46:22.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:21 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2797722501' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:22.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:21 trial151 ceph-mon[39453]: pgmap v199: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 755 KiB/s rd, 7.1 KiB/s wr, 685 op/s 2026-02-24T16:46:22.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:21 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2797722501' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:22.733 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:22 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:22] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:37726 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode2: 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout:NQN NSID Bdev RBD Mode Image Block UUID Load Location Visibility Encryption IOs per R/W, R, W MBs 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout: Name Image Size Size Balancing second per second 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout: Group 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 1 bdev_33263a91-b0f5- mypool/myimage21 Read-Write 8 GiB 512 Bytes 33263a91-b0f5-4879- 2 All Hosts None unset unset 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout: 4879-8060-11a137a1edea 8060-11a137a1edea unset 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.005 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 2 bdev_d78cb7f3-762d- mypool/myimage22 Read-Write 8 GiB 512 Bytes d78cb7f3-762d-40e2- 3 All Hosts None unset unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: 40e2-abb5-ef80b2267e84 abb5-ef80b2267e84 unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 3 bdev_697843d5-c53e- mypool/myimage23 Read-Write 8 GiB 512 Bytes 697843d5-c53e-4486- 4 All Hosts None unset unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: 4486-a787-6dbdf2e25dcc a787-6dbdf2e25dcc unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 4 bdev_bf2e4988-73d0- mypool/myimage24 Read-Write 8 GiB 512 Bytes bf2e4988-73d0-431b- 1 All Hosts None unset unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: 431b-82a3-e4b3011875b2 82a3-e4b3011875b2 unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 5 bdev_d59628d5-17f8- mypool/myimage25 Read-Write 8 GiB 512 Bytes d59628d5-17f8-4bf7- 2 All Hosts None unset unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: 4bf7-a126-95d793d0a138 a126-95d793d0a138 unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.006 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 6 bdev_85b21803-9ff2- mypool/myimage26 Read-Write 8 GiB 512 Bytes 85b21803-9ff2-4dc2- 3 All Hosts None unset unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: 4dc2-bebf-ba6ba79bb040 bebf-ba6ba79bb040 unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 7 bdev_4919133d-7d04- mypool/myimage27 Read-Write 8 GiB 512 Bytes 4919133d-7d04-4f44- 4 All Hosts None unset unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: 4f44-8d58-32e340ab9034 8d58-32e340ab9034 unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 8 bdev_47256c90-d0c1- mypool/myimage28 Read-Write 8 GiB 512 Bytes 47256c90-d0c1-47e8- 1 All Hosts None unset unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: 47e8-822d-cea30bc74f24 822d-cea30bc74f24 unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 9 bdev_1955cd27-99fd- mypool/myimage29 Read-Write 8 GiB 512 Bytes 1955cd27-99fd-4931- 2 All Hosts None unset unset 2026-02-24T16:46:23.007 INFO:tasks.workunit.client.1.trial052.stdout: 4931-a07c-9ddaafcfa3f9 a07c-9ddaafcfa3f9 unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 10 bdev_561ccdee-bf1f- mypool/myimage30 Read-Write 8 GiB 512 Bytes 561ccdee-bf1f-4cee- 3 All Hosts None unset unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: 4cee-bfc7-b59a062726db bfc7-b59a062726db unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 11 bdev_e58f322b-54d4- mypool/myimage31 Read-Write 8 GiB 512 Bytes e58f322b-54d4-498a- 4 All Hosts None unset unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: 498a-9468-0145cb626612 9468-0145cb626612 unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 12 bdev_39df5daa-d0fe- mypool/myimage32 Read-Write 8 GiB 512 Bytes 39df5daa-d0fe-46f3- 1 All Hosts None unset unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: 46f3-87cd-17f324297665 87cd-17f324297665 unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 13 bdev_3aeacf00-aa1c- mypool/myimage33 Read-Write 8 GiB 512 Bytes 3aeacf00-aa1c-433c- 2 All Hosts None unset unset 2026-02-24T16:46:23.008 INFO:tasks.workunit.client.1.trial052.stdout: 433c-9d46-b94cdf90b730 9d46-b94cdf90b730 unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 14 bdev_3fd1d7af-bc41- mypool/myimage34 Read-Write 8 GiB 512 Bytes 3fd1d7af-bc41-4ce5- 3 All Hosts None unset unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: 4ce5-be3b-421cfd63d96c be3b-421cfd63d96c unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 15 bdev_1c4ecda8-6e7f- mypool/myimage35 Read-Write 8 GiB 512 Bytes 1c4ecda8-6e7f-4b9c- 4 All Hosts None unset unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: 4b9c-8a6b-d2f9cd6a8229 8a6b-d2f9cd6a8229 unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 16 bdev_d6b120df-b3ac- mypool/myimage36 Read-Write 8 GiB 512 Bytes d6b120df-b3ac-4704- 1 All Hosts None unset unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: 4704-843a-d114552783c9 843a-d114552783c9 unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 17 bdev_f2b63888-78d5- mypool/myimage37 Read-Write 8 GiB 512 Bytes f2b63888-78d5-427e- 2 All Hosts None unset unset 2026-02-24T16:46:23.009 INFO:tasks.workunit.client.1.trial052.stdout: 427e-986d-35708fc88a7e 986d-35708fc88a7e unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 18 bdev_e7481463-7d31- mypool/myimage38 Read-Write 8 GiB 512 Bytes e7481463-7d31-4d0d- 3 All Hosts None unset unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: 4d0d-84ee-837a0ee299d5 84ee-837a0ee299d5 unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 19 bdev_7c5fad51-f463- mypool/myimage39 Read-Write 8 GiB 512 Bytes 7c5fad51-f463-4927- 4 All Hosts None unset unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: 4927-bfdb-c5c90e7c7645 bfdb-c5c90e7c7645 unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 20 bdev_95b5e395-9997- mypool/myimage40 Read-Write 8 GiB 512 Bytes 95b5e395-9997-489f- 1 All Hosts None unset unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: 489f-8ead-5f6a6bdae442 8ead-5f6a6bdae442 unset 2026-02-24T16:46:23.010 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 21 bdev_7f04c0eb-d018- mypool/test6 Read-Write 8 GiB 512 Bytes 7f04c0eb-d018-4904- 3 All Hosts None unset unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: 4904-938d-15e96a4242ff 938d-15e96a4242ff unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 22 bdev_2c5b330d-721e- mypool/test7 Read-Write 8 GiB 512 Bytes 2c5b330d-721e-4e7c- 4 All Hosts None unset unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: 4e7c-b2bc-b2fdf32e4d01 b2bc-b2fdf32e4d01 unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 23 bdev_89556831-7eca- mypool/test8 Read-Write 8 GiB 512 Bytes 89556831-7eca-4c4a- 1 All Hosts None unset unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: 4c4a-8c05-d6cd69affbf6 8c05-d6cd69affbf6 unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 24 bdev_40d6f514-4245- mypool/test9 Read-Write 8 GiB 512 Bytes 40d6f514-4245-4aa7- 2 All Hosts None unset unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: 4aa7-b64b-eef850664928 b64b-eef850664928 unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.011 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode2 25 bdev_adef4f3b-3318- mypool/test10 Read-Write 8 GiB 512 Bytes adef4f3b-3318-4990- 3 All Hosts None unset unset 2026-02-24T16:46:23.012 INFO:tasks.workunit.client.1.trial052.stdout: 4990-b2d1-88e6364bf3f3 b2d1-88e6364bf3f3 unset 2026-02-24T16:46:23.012 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:23.024 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:23.024 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:23.024 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:23.062 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:23Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:23.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4116008026' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:23.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4116008026' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:23.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4116008026' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:23.788 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:23 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:23] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:51008 2026-02-24T16:46:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:23 trial153 ceph-mon[44017]: pgmap v200: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 7.0 KiB/s wr, 1.01k op/s 2026-02-24T16:46:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1393811763' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:24.131 INFO:tasks.workunit.client.1.trial052.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode3: 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout:NQN NSID Bdev RBD Mode Image Block UUID Load Location Visibility Encryption IOs per R/W, R, W MBs 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: Name Image Size Size Balancing second per second 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: Group 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 1 bdev_d5afe5a9-1637- mypool/myimage41 Read-Write 8 GiB 512 Bytes d5afe5a9-1637-4f3b- 2 All Hosts None unset unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: 4f3b-b953-fb9578bad99f b953-fb9578bad99f unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 2 bdev_8b96a726-7bf9- mypool/myimage42 Read-Write 8 GiB 512 Bytes 8b96a726-7bf9-4da5- 3 All Hosts None unset unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: 4da5-9239-52b473dec480 9239-52b473dec480 unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.132 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 3 bdev_28c3b9d1-254e- mypool/myimage43 Read-Write 8 GiB 512 Bytes 28c3b9d1-254e-42d7- 4 All Hosts None unset unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: 42d7-86d2-bd3ecc9709f3 86d2-bd3ecc9709f3 unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 4 bdev_3a5dd3f8-cf76- mypool/myimage44 Read-Write 8 GiB 512 Bytes 3a5dd3f8-cf76-4fe9- 1 All Hosts None unset unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: 4fe9-afb4-e6b457fa19d6 afb4-e6b457fa19d6 unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 5 bdev_7e2595e1-3547- mypool/myimage45 Read-Write 8 GiB 512 Bytes 7e2595e1-3547-4c9f- 2 All Hosts None unset unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: 4c9f-8c23-62f4df57a5bd 8c23-62f4df57a5bd unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 6 bdev_8309032c-1aea- mypool/myimage46 Read-Write 8 GiB 512 Bytes 8309032c-1aea-458a- 3 All Hosts None unset unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: 458a-8fd6-d156a3fd0e03 8fd6-d156a3fd0e03 unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.133 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 7 bdev_c98b8285-98f9- mypool/myimage47 Read-Write 8 GiB 512 Bytes c98b8285-98f9-4f4f- 4 All Hosts None unset unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: 4f4f-9dd5-5a800f1276e1 9dd5-5a800f1276e1 unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 8 bdev_04c36010-6526- mypool/myimage48 Read-Write 8 GiB 512 Bytes 04c36010-6526-485a- 1 All Hosts None unset unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: 485a-9329-6916b03ccedb 9329-6916b03ccedb unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 9 bdev_e44b0407-8e5a- mypool/myimage49 Read-Write 8 GiB 512 Bytes e44b0407-8e5a-4024- 2 All Hosts None unset unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: 4024-9fd0-d27c03edf6a8 9fd0-d27c03edf6a8 unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 10 bdev_142f883b-d314- mypool/myimage50 Read-Write 8 GiB 512 Bytes 142f883b-d314-4f53- 3 All Hosts None unset unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: 4f53-aaf8-d56e636cc5e8 aaf8-d56e636cc5e8 unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 11 bdev_1c7e3336-092f- mypool/myimage51 Read-Write 8 GiB 512 Bytes 1c7e3336-092f-4916- 4 All Hosts None unset unset 2026-02-24T16:46:24.134 INFO:tasks.workunit.client.1.trial052.stdout: 4916-bba8-0f00dd7b20bd bba8-0f00dd7b20bd unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 12 bdev_b0854e12-150b- mypool/myimage52 Read-Write 8 GiB 512 Bytes b0854e12-150b-430e- 1 All Hosts None unset unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: 430e-98fb-f47be6a0b321 98fb-f47be6a0b321 unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 13 bdev_608a7ae4-773e- mypool/myimage53 Read-Write 8 GiB 512 Bytes 608a7ae4-773e-4ef4- 2 All Hosts None unset unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: 4ef4-907e-7149bcc633ca 907e-7149bcc633ca unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 14 bdev_7b358c27-f143- mypool/myimage54 Read-Write 8 GiB 512 Bytes 7b358c27-f143-44e8- 3 All Hosts None unset unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: 44e8-a648-5edbc32d6d46 a648-5edbc32d6d46 unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 15 bdev_4e5113f1-ae3e- mypool/myimage55 Read-Write 8 GiB 512 Bytes 4e5113f1-ae3e-4036- 4 All Hosts None unset unset 2026-02-24T16:46:24.135 INFO:tasks.workunit.client.1.trial052.stdout: 4036-9ede-fd1641d29daa 9ede-fd1641d29daa unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 16 bdev_4c615bb7-4edb- mypool/myimage56 Read-Write 8 GiB 512 Bytes 4c615bb7-4edb-4cc1- 1 All Hosts None unset unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: 4cc1-92b5-3c599225e672 92b5-3c599225e672 unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 17 bdev_3c219351-ecdb- mypool/myimage57 Read-Write 8 GiB 512 Bytes 3c219351-ecdb-43be- 2 All Hosts None unset unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: 43be-a263-14ffde10208a a263-14ffde10208a unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 18 bdev_99635a96-175a- mypool/myimage58 Read-Write 8 GiB 512 Bytes 99635a96-175a-4876- 3 All Hosts None unset unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: 4876-9a62-b370d6a4fe90 9a62-b370d6a4fe90 unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 19 bdev_8674d792-6821- mypool/myimage59 Read-Write 8 GiB 512 Bytes 8674d792-6821-465c- 4 All Hosts None unset unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: 465c-bc6a-a7cb2de67dd8 bc6a-a7cb2de67dd8 unset 2026-02-24T16:46:24.136 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 20 bdev_6b2cbf76-d023- mypool/myimage60 Read-Write 8 GiB 512 Bytes 6b2cbf76-d023-4a57- 1 All Hosts None unset unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: 4a57-a654-b59345cf9a2a a654-b59345cf9a2a unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 21 bdev_de5fbfd4-f25e- mypool/test11 Read-Write 8 GiB 512 Bytes de5fbfd4-f25e-4063- 4 All Hosts None unset unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: 4063-8285-f991bd3771cf 8285-f991bd3771cf unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 22 bdev_29461d02-252c- mypool/test12 Read-Write 8 GiB 512 Bytes 29461d02-252c-42e1- 1 All Hosts None unset unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: 42e1-8f87-67b48090adee 8f87-67b48090adee unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 23 bdev_8c20d236-b6aa- mypool/test13 Read-Write 8 GiB 512 Bytes 8c20d236-b6aa-48fc- 2 All Hosts None unset unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: 48fc-a6a4-f5b39a572d63 a6a4-f5b39a572d63 unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.137 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 24 bdev_987771c0-d05d- mypool/test14 Read-Write 8 GiB 512 Bytes 987771c0-d05d-4b2a- 3 All Hosts None unset unset 2026-02-24T16:46:24.138 INFO:tasks.workunit.client.1.trial052.stdout: 4b2a-9f0b-eeca599a2c96 9f0b-eeca599a2c96 unset 2026-02-24T16:46:24.138 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.138 INFO:tasks.workunit.client.1.trial052.stdout:nqn.2016-06.io.spdk:cnode3 25 bdev_962a91c0-43ce- mypool/test15 Read-Write 8 GiB 512 Bytes 962a91c0-43ce-4a6d- 4 All Hosts None unset unset 2026-02-24T16:46:24.138 INFO:tasks.workunit.client.1.trial052.stdout: 4a6d-8a89-ff002b365111 8a89-ff002b365111 unset 2026-02-24T16:46:24.138 INFO:tasks.workunit.client.1.trial052.stdout: unset 2026-02-24T16:46:24.151 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count_per_subsys=25 2026-02-24T16:46:24.151 INFO:tasks.workunit.client.1.trial052.stderr:+ assert_namespaces_count 25 2026-02-24T16:46:24.151 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count_per_subsys=25 2026-02-24T16:46:24.152 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json subsystem list 2026-02-24T16:46:24.152 INFO:tasks.workunit.client.1.trial052.stderr:++ grep namespace_count 2026-02-24T16:46:24.152 INFO:tasks.workunit.client.1.trial052.stderr:++ grep 25 2026-02-24T16:46:24.152 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:46:24.191 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:24Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:24.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:23 trial183 ceph-mon[43553]: pgmap v200: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 7.0 KiB/s wr, 1.01k op/s 2026-02-24T16:46:24.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1393811763' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:24.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:23 trial151 ceph-mon[39453]: pgmap v200: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 7.0 KiB/s wr, 1.01k op/s 2026-02-24T16:46:24.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1393811763' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:24.679 INFO:tasks.workunit.client.1.trial052.stderr:+ actual_count=3 2026-02-24T16:46:24.680 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 3 -ne 3 ']' 2026-02-24T16:46:24.680 INFO:tasks.workunit.client.1.trial052.stderr:++ seq 1 3 2026-02-24T16:46:24.681 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:24.681 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:24.681 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json namespace list --subsystem nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:24.681 INFO:tasks.workunit.client.1.trial052.stderr:++ jq -r '.namespaces[] | select(.rbd_image_name | startswith("test")) | .nsid' 2026-02-24T16:46:24.719 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:24Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:24.983 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:24 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:24] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.52:43080 2026-02-24T16:46:25.483 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:25 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:25] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:57494 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:+ NSIDs='21 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:22 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:23 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:24 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:25' 2026-02-24T16:46:25.801 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:25.802 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode1 --nsid 21 2026-02-24T16:46:25.841 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:25Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:26.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:25 trial153 ceph-mon[44017]: pgmap v201: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:26.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:25 trial183 ceph-mon[43553]: pgmap v201: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:26.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:25 trial151 ceph-mon[39453]: pgmap v201: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:26.405 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 21 from nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:26.431 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:26.431 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode1 --nsid 22 2026-02-24T16:46:26.466 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:26.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:26.405958] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:26.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:26.406004] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:26.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:26.401579] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:26.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:26.401625] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:26.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:42574 2026-02-24T16:46:26.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode1, client address: IPv4 10.20.193.52:42574 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:26.399553] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:26.399594] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:26.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO grpc.py:1773 (2): Received request to delete bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5, client address: IPv4 10.20.193.52:42574 2026-02-24T16:46:26.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:26.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:26.402015] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:26.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:26.402058] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.034 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 22 from nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:27.058 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:27.058 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode1 --nsid 23 2026-02-24T16:46:27.096 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:27.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO state.py:1684 (105): Start update from 94 to 97 (discovery-trial183). 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO state.py:2068 (105): Update complete (94 -> 97) (discovery-trial183). 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO state.py:1684 (2): Start update from 94 to 97 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test1, no need to remove 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO grpc.py:1773 (2): Received request to delete bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] INFO state.py:2068 (2): Update complete (94 -> 97) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:27.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:26 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:27.029884] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:27.029922] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:1684 (81): Start update from 94 to 97 (discovery-trial151). 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:2068 (81): Update complete (94 -> 97) (discovery-trial151). 2026-02-24T16:46:27.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:58330 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode1, client address: IPv4 10.20.193.52:58330 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:26 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:26] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:27.027740] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:27.027781] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO grpc.py:1773 (2): Received request to delete bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6, client address: IPv4 10.20.193.52:58330 2026-02-24T16:46:27.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO state.py:1684 (2): Start update from 94 to 97 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:27.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO state.py:1684 (132): Start update from 94 to 97 (discovery-trial153). 2026-02-24T16:46:27.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO state.py:2068 (132): Update complete (94 -> 97) (discovery-trial153). 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test1, no need to remove 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO grpc.py:1773 (2): Received request to delete bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] INFO state.py:2068 (2): Update complete (94 -> 97) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:26 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:26] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:27.030432] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:27.030471] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO state.py:1684 (126): Start update from 94 to 97 (discovery-trial191). 2026-02-24T16:46:27.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO state.py:2068 (126): Update complete (94 -> 97) (discovery-trial191). 2026-02-24T16:46:27.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO state.py:1684 (2): Start update from 94 to 97 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:27.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test1, no need to remove 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO grpc.py:1773 (2): Received request to delete bdev bdev_83e00b6e-24fd-4352-8e44-3a7f06bf26c5 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] INFO state.py:2068 (2): Update complete (94 -> 97) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:26 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:26] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:27.034269] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:27.034312] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.655 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 23 from nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:27.676 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:27.676 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode1 --nsid 24 2026-02-24T16:46:27.725 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:27.733 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO state.py:1684 (105): Start update from 97 to 100 (discovery-trial183). 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO state.py:2068 (105): Update complete (97 -> 100) (discovery-trial183). 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO state.py:1684 (2): Start update from 97 to 100 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test2, no need to remove 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO grpc.py:1773 (2): Received request to delete bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] INFO state.py:2068 (2): Update complete (97 -> 100) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:27] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:27.651771] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:27.651802] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO state.py:1684 (81): Start update from 97 to 100 (discovery-trial151). 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO state.py:2068 (81): Update complete (97 -> 100) (discovery-trial151). 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:57266 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode1, client address: IPv4 10.20.193.52:57266 2026-02-24T16:46:27.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:27.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:27.649673] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:27.649712] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:27] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b, client address: IPv4 10.20.193.52:57266 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO state.py:1684 (2): Start update from 97 to 100 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO state.py:1684 (132): Start update from 97 to 100 (discovery-trial153). 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO state.py:2068 (132): Update complete (97 -> 100) (discovery-trial153). 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test2, no need to remove 2026-02-24T16:46:27.793 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO grpc.py:1773 (2): Received request to delete bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 2026-02-24T16:46:27.794 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] INFO state.py:2068 (2): Update complete (97 -> 100) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:27.794 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.794 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.794 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:27.652179] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.794 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:27.652216] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO state.py:1684 (2): Start update from 97 to 100 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO state.py:1684 (126): Start update from 97 to 100 (discovery-trial191). 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO state.py:2068 (126): Update complete (97 -> 100) (discovery-trial191). 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test2, no need to remove 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO grpc.py:1773 (2): Received request to delete bdev bdev_b6118346-75e6-43bd-bf7c-1f36e66325d6 2026-02-24T16:46:27.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] INFO state.py:2068 (2): Update complete (97 -> 100) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:27.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:27] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:27.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:27.656300] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:27.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:27 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:27.656335] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:27 trial153 ceph-mon[44017]: pgmap v202: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.16k op/s 2026-02-24T16:46:28.097 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:27 trial183 ceph-mon[43553]: pgmap v202: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.16k op/s 2026-02-24T16:46:28.101 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:27 trial151 ceph-mon[39453]: pgmap v202: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 1.16k op/s 2026-02-24T16:46:28.293 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 24 from nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:28.314 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:28.314 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode1 --nsid 25 2026-02-24T16:46:28.346 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:28.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 100 to 103 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:1684 (105): Start update from 100 to 103 (discovery-trial183). 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:2068 (105): Update complete (100 -> 103) (discovery-trial183). 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test3, no need to remove 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (100 -> 103) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:28.290187] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:28.290227] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:1684 (81): Start update from 100 to 103 (discovery-trial151). 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:2068 (81): Update complete (100 -> 103) (discovery-trial151). 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:51740 2026-02-24T16:46:28.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode1, client address: IPv4 10.20.193.52:51740 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:28.287871] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:28.287911] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:28.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261, client address: IPv4 10.20.193.52:51740 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:1684 (132): Start update from 100 to 103 (discovery-trial153). 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:2068 (132): Update complete (100 -> 103) (discovery-trial153). 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 100 to 103 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test3, no need to remove 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b 2026-02-24T16:46:28.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (100 -> 103) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:28.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:28.290462] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:28.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:28.290566] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:1684 (126): Start update from 100 to 103 (discovery-trial191). 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:2068 (126): Update complete (100 -> 103) (discovery-trial191). 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 100 to 103 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test3, no need to remove 2026-02-24T16:46:28.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7491df4d-8f09-4e64-a6b1-9e16d03ee72b 2026-02-24T16:46:28.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (100 -> 103) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:28.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:28.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:28.294739] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:28.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:28.294782] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:28.904 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 25 from nqn.2016-06.io.spdk:cnode1: Successful 2026-02-24T16:46:28.926 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:28.926 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:28.927 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json namespace list --subsystem nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:28.927 INFO:tasks.workunit.client.1.trial052.stderr:++ jq -r '.namespaces[] | select(.rbd_image_name | startswith("test")) | .nsid' 2026-02-24T16:46:28.964 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:29.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2937409776' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:29.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 103 to 106 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:29.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.063 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:1684 (132): Start update from 103 to 106 (discovery-trial153). 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:2068 (132): Update complete (103 -> 106) (discovery-trial153). 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test4, no need to remove 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (103 -> 106) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:28] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:28.901119] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:29.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:28 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:28.901158] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:29.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 103 to 106 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:1684 (126): Start update from 103 to 106 (discovery-trial191). 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:2068 (126): Update complete (103 -> 106) (discovery-trial191). 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test4, no need to remove 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (103 -> 106) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:28] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:28.905339] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:29.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:28 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:28.905377] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:29.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2937409776' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:29.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:1684 (2): Start update from 103 to 106 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:29.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:1684 (105): Start update from 103 to 106 (discovery-trial183). 2026-02-24T16:46:29.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:2068 (105): Update complete (103 -> 106) (discovery-trial183). 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test4, no need to remove 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f38fa755-e994-4f4a-befe-4412b4fca261 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] INFO state.py:2068 (2): Update complete (103 -> 106) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:28] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:28.900415] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:29.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:28 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:28.900453] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:29.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2937409776' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:29.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:1684 (81): Start update from 103 to 106 (discovery-trial151). 2026-02-24T16:46:29.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:2068 (81): Update complete (103 -> 106) (discovery-trial151). 2026-02-24T16:46:29.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 10.20.193.52:39126 2026-02-24T16:46:29.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:29.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode1, client address: IPv4 10.20.193.52:39126 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:28.898300] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:28.898339] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:29.237 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:28 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:28] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410, client address: IPv4 10.20.193.52:39126 2026-02-24T16:46:29.733 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:29] INFO state.py:1684 (81): Start update from 106 to 109 (discovery-trial151). 2026-02-24T16:46:29.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:29] INFO state.py:2068 (81): Update complete (106 -> 109) (discovery-trial151). 2026-02-24T16:46:29.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:29] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:35198 2026-02-24T16:46:29.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO state.py:1684 (2): Start update from 106 to 109 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO state.py:1684 (105): Start update from 106 to 109 (discovery-trial183). 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO state.py:2068 (105): Update complete (106 -> 109) (discovery-trial183). 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test5, no need to remove 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 2026-02-24T16:46:29.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:29] INFO state.py:2068 (2): Update complete (106 -> 109) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO state.py:1684 (2): Start update from 106 to 109 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO state.py:1684 (132): Start update from 106 to 109 (discovery-trial153). 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO state.py:2068 (132): Update complete (106 -> 109) (discovery-trial153). 2026-02-24T16:46:29.798 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test5, no need to remove 2026-02-24T16:46:29.799 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 2026-02-24T16:46:29.799 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:29] INFO state.py:2068 (2): Update complete (106 -> 109) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO state.py:1684 (2): Start update from 106 to 109 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode1, I am sure, context: None 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode1 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO state.py:1684 (126): Start update from 106 to 109 (discovery-trial191). 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO state.py:2068 (126): Update complete (106 -> 109) (discovery-trial191). 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test5, no need to remove 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO grpc.py:1773 (2): Received request to delete bdev bdev_f4382a38-b1e9-4be8-8684-47dd99dff410 2026-02-24T16:46:29.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:29 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:29] INFO state.py:2068 (2): Update complete (106 -> 109) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-mon[44017]: pgmap v203: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 864 op/s 2026-02-24T16:46:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2102588844' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1842617490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.067 INFO:tasks.workunit.client.1.trial052.stderr:+ NSIDs='21 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:22 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:23 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:24 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:25' 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:30.068 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode2 --nsid 21 2026-02-24T16:46:30.096 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:30Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:30.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-mon[43553]: pgmap v203: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 864 op/s 2026-02-24T16:46:30.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2102588844' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1842617490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-mon[39453]: pgmap v203: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.1 MiB/s rd, 3.2 KiB/s wr, 864 op/s 2026-02-24T16:46:30.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2102588844' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1842617490' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.622 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 21 from nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:30.641 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:30.642 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode2 --nsid 22 2026-02-24T16:46:30.681 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:30Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:30.803 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:45838 2026-02-24T16:46:30.803 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:30.803 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode2, client address: IPv4 10.20.193.52:45838 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:30.616723] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:30.616761] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:30.804 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:30] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff, client address: IPv4 10.20.193.52:45838 2026-02-24T16:46:30.813 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2388968313' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:30] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:30.619609] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:30.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:30 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:30.619644] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:30.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:30] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:30.623690] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:30.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:30 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:30.623729] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:30.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2388968313' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:30] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:30] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:30.619356] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:30.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:30 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:30.619400] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.070 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2388968313' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:31.257 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 22 from nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:31.279 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:31.280 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode2 --nsid 23 2026-02-24T16:46:31.323 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:1684 (126): Start update from 109 to 112 (discovery-trial191). 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 109 to 112 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:2068 (126): Update complete (109 -> 112) (discovery-trial191). 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test6, no need to remove 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff 2026-02-24T16:46:31.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (109 -> 112) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:31.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:31.258682] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:31.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:31.258717] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 109 to 112 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:1684 (105): Start update from 109 to 112 (discovery-trial183). 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:2068 (105): Update complete (109 -> 112) (discovery-trial183). 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test6, no need to remove 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (109 -> 112) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:31.254122] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:31.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:31.254162] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:1684 (81): Start update from 109 to 112 (discovery-trial151). 2026-02-24T16:46:31.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:2068 (81): Update complete (109 -> 112) (discovery-trial151). 2026-02-24T16:46:31.485 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:53552 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode2, client address: IPv4 10.20.193.52:53552 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:31.251618] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:31.251656] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01, client address: IPv4 10.20.193.52:53552 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 109 to 112 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:1684 (132): Start update from 109 to 112 (discovery-trial153). 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:2068 (132): Update complete (109 -> 112) (discovery-trial153). 2026-02-24T16:46:31.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test6, no need to remove 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_7f04c0eb-d018-4904-938d-15e96a4242ff 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (109 -> 112) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:31.254263] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:31.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:31.254302] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:1684 (105): Start update from 112 to 115 (discovery-trial183). 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 112 to 115 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:2068 (105): Update complete (112 -> 115) (discovery-trial183). 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:31.749 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test7, no need to remove 2026-02-24T16:46:31.883 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 23 from nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:31.905 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:31.905 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode2 --nsid 24 2026-02-24T16:46:31.951 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:31.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-mon[39453]: pgmap v204: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 892 KiB/s rd, 1.9 KiB/s wr, 806 op/s 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:1684 (81): Start update from 112 to 115 (discovery-trial151). 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:2068 (81): Update complete (112 -> 115) (discovery-trial151). 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:34882 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode2, client address: IPv4 10.20.193.52:34882 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:31.878458] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:31.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:31.878492] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:31.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:31 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6, client address: IPv4 10.20.193.52:34882 2026-02-24T16:46:32.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-mon[44017]: pgmap v204: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 892 KiB/s rd, 1.9 KiB/s wr, 806 op/s 2026-02-24T16:46:32.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 112 to 115 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:32.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:32.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:32.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:1684 (132): Start update from 112 to 115 (discovery-trial153). 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:2068 (132): Update complete (112 -> 115) (discovery-trial153). 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test7, no need to remove 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (112 -> 115) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:31] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:31.881094] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:31 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:31.881135] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:1684 (2): Start update from 112 to 115 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:32.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:32.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:1684 (126): Start update from 112 to 115 (discovery-trial191). 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:2068 (126): Update complete (112 -> 115) (discovery-trial191). 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test7, no need to remove 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (112 -> 115) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:31] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:31.885537] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.093 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:31 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:31.885571] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-mon[43553]: pgmap v204: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 892 KiB/s rd, 1.9 KiB/s wr, 806 op/s 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO grpc.py:1773 (2): Received request to delete bdev bdev_2c5b330d-721e-4e7c-b2bc-b2fdf32e4d01 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] INFO state.py:2068 (2): Update complete (112 -> 115) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:31] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:31.880841] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:31 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:31.880875] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.522 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 24 from nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:32.549 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:32.549 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode2 --nsid 25 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 115 to 118 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:1684 (126): Start update from 115 to 118 (discovery-trial191). 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:2068 (126): Update complete (115 -> 118) (discovery-trial191). 2026-02-24T16:46:32.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test8, no need to remove 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO grpc.py:1773 (2): Received request to delete bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:2068 (2): Update complete (115 -> 118) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:32.522614] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:32.522650] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.592 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:32Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:1684 (105): Start update from 115 to 118 (discovery-trial183). 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 115 to 118 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:2068 (105): Update complete (115 -> 118) (discovery-trial183). 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test8, no need to remove 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO grpc.py:1773 (2): Received request to delete bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 2026-02-24T16:46:32.734 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:2068 (2): Update complete (115 -> 118) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:32.518427] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:32.518459] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:1684 (81): Start update from 115 to 118 (discovery-trial151). 2026-02-24T16:46:32.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:2068 (81): Update complete (115 -> 118) (discovery-trial151). 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:35684 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode2, client address: IPv4 10.20.193.52:35684 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:32.516125] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:32.516161] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:32.736 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO grpc.py:1773 (2): Received request to delete bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928, client address: IPv4 10.20.193.52:35684 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 115 to 118 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:1684 (132): Start update from 115 to 118 (discovery-trial153). 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:2068 (132): Update complete (115 -> 118) (discovery-trial153). 2026-02-24T16:46:32.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test8, no need to remove 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO grpc.py:1773 (2): Received request to delete bdev bdev_89556831-7eca-4c4a-8c05-d6cd69affbf6 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:2068 (2): Update complete (115 -> 118) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:32.518782] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:32.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:32.518820] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:33.164 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 25 from nqn.2016-06.io.spdk:cnode2: Successful 2026-02-24T16:46:33.185 INFO:tasks.workunit.client.1.trial052.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2026-02-24T16:46:33.185 INFO:tasks.workunit.client.1.trial052.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:33.186 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json namespace list --subsystem nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:33.186 INFO:tasks.workunit.client.1.trial052.stderr:++ jq -r '.namespaces[] | select(.rbd_image_name | startswith("test")) | .nsid' 2026-02-24T16:46:33.225 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:33Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:33.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 118 to 121 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:33.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:1684 (105): Start update from 118 to 121 (discovery-trial183). 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:32 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:32] INFO state.py:2068 (105): Update complete (118 -> 121) (discovery-trial183). 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test9, no need to remove 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (118 -> 121) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:33.161104] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:33.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:33.161138] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:1684 (81): Start update from 118 to 121 (discovery-trial151). 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:32 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:32] INFO state.py:2068 (81): Update complete (118 -> 121) (discovery-trial151). 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 10.20.193.52:55192 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode2, client address: IPv4 10.20.193.52:55192 2026-02-24T16:46:33.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:33.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:33.158743] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:33.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:33.158779] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:33.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3, client address: IPv4 10.20.193.52:55192 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 118 to 121 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:1684 (132): Start update from 118 to 121 (discovery-trial153). 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:32] INFO state.py:2068 (132): Update complete (118 -> 121) (discovery-trial153). 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test9, no need to remove 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 2026-02-24T16:46:33.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (118 -> 121) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:33.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:33.161485] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:33.315 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:33.161526] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:1684 (126): Start update from 118 to 121 (discovery-trial191). 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:2068 (126): Update complete (118 -> 121) (discovery-trial191). 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO state.py:1684 (2): Start update from 118 to 121 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:32 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:32] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test9, no need to remove 2026-02-24T16:46:33.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_40d6f514-4245-4aa7-b64b-eef850664928 2026-02-24T16:46:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (118 -> 121) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:33.165724] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:33.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:33.165759] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-mon[43553]: pgmap v205: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:33.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO state.py:1684 (2): Start update from 121 to 124 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:33.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:33.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:33.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO state.py:1684 (105): Start update from 121 to 124 (discovery-trial183). 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO state.py:2068 (105): Update complete (121 -> 124) (discovery-trial183). 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test10, no need to remove 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:33 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (121 -> 124) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-mon[39453]: pgmap v205: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO state.py:1684 (81): Start update from 121 to 124 (discovery-trial151). 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO state.py:2068 (81): Update complete (121 -> 124) (discovery-trial151). 2026-02-24T16:46:33.984 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:33 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:33] INFO grpc.py:3790 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:50384 2026-02-24T16:46:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-mon[44017]: pgmap v205: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.2 KiB/s wr, 984 op/s 2026-02-24T16:46:34.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO state.py:1684 (132): Start update from 121 to 124 (discovery-trial153). 2026-02-24T16:46:34.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO state.py:2068 (132): Update complete (121 -> 124) (discovery-trial153). 2026-02-24T16:46:34.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO state.py:1684 (2): Start update from 121 to 124 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:34.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:34.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:34.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test10, no need to remove 2026-02-24T16:46:34.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 2026-02-24T16:46:34.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:33 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (121 -> 124) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO state.py:1684 (126): Start update from 121 to 124 (discovery-trial191). 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO state.py:2068 (126): Update complete (121 -> 124) (discovery-trial191). 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO state.py:1684 (2): Start update from 121 to 124 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode2, I am sure, context: None 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode2 2026-02-24T16:46:34.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test10, no need to remove 2026-02-24T16:46:34.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO grpc.py:1773 (2): Received request to delete bdev bdev_adef4f3b-3318-4990-b2d1-88e6364bf3f3 2026-02-24T16:46:34.092 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:33 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:33] INFO state.py:2068 (2): Update complete (121 -> 124) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:34.258 INFO:tasks.workunit.client.1.trial052.stderr:+ NSIDs='21 2026-02-24T16:46:34.258 INFO:tasks.workunit.client.1.trial052.stderr:22 2026-02-24T16:46:34.258 INFO:tasks.workunit.client.1.trial052.stderr:23 2026-02-24T16:46:34.258 INFO:tasks.workunit.client.1.trial052.stderr:24 2026-02-24T16:46:34.258 INFO:tasks.workunit.client.1.trial052.stderr:25' 2026-02-24T16:46:34.259 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:34.259 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode3 --nsid 21 2026-02-24T16:46:34.285 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:34Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:34.869 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 21 from nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:34.889 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:34.889 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode3 --nsid 22 2026-02-24T16:46:34.928 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:34Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:35.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:34] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:34.865798] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:34 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:34.865833] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.090 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:34] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:34.870005] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:34 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:34.870040] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.233 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:34] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:34.865214] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:34 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:34.865257] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.234 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:52482 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode3, client address: IPv4 10.20.193.52:52482 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:34.863344] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:34.863380] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:34 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:34] INFO grpc.py:1773 (2): Received request to delete bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf, client address: IPv4 10.20.193.52:52482 2026-02-24T16:46:35.511 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 22 from nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:35.534 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:35.534 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode3 --nsid 23 2026-02-24T16:46:35.563 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:1684 (132): Start update from 124 to 127 (discovery-trial153). 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:2068 (132): Update complete (124 -> 127) (discovery-trial153). 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 124 to 127 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test11, no need to remove 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO grpc.py:1773 (2): Received request to delete bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:2068 (2): Update complete (124 -> 127) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:35.507512] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:35.507553] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.567 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:35Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:1684 (126): Start update from 124 to 127 (discovery-trial191). 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:2068 (126): Update complete (124 -> 127) (discovery-trial191). 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 124 to 127 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test11, no need to remove 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO grpc.py:1773 (2): Received request to delete bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf 2026-02-24T16:46:35.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:2068 (2): Update complete (124 -> 127) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:35.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:35.511802] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:35.511841] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:1684 (81): Start update from 124 to 127 (discovery-trial151). 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:2068 (81): Update complete (124 -> 127) (discovery-trial151). 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:58198 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode3, client address: IPv4 10.20.193.52:58198 2026-02-24T16:46:35.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:35.504934] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:35.504971] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO grpc.py:1773 (2): Received request to delete bdev bdev_29461d02-252c-42e1-8f87-67b48090adee, client address: IPv4 10.20.193.52:58198 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:1684 (105): Start update from 124 to 127 (discovery-trial183). 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:2068 (105): Update complete (124 -> 127) (discovery-trial183). 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 124 to 127 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 21 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:35.735 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 21 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test11, no need to remove 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO grpc.py:1773 (2): Received request to delete bdev bdev_de5fbfd4-f25e-4063-8285-f991bd3771cf 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:2068 (2): Update complete (124 -> 127) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:35.507408] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:35.736 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:35.507449] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.095 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-mon[39453]: pgmap v206: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 897 KiB/s rd, 2.0 KiB/s wr, 631 op/s 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:1684 (81): Start update from 127 to 130 (discovery-trial151). 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:35 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:35] INFO state.py:2068 (81): Update complete (127 -> 130) (discovery-trial151). 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:54824 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode3, client address: IPv4 10.20.193.52:54824 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:36.099026] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:36.099062] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.097 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-mon[44017]: pgmap v206: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 897 KiB/s rd, 2.0 KiB/s wr, 631 op/s 2026-02-24T16:46:36.097 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-mon[43553]: pgmap v206: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 897 KiB/s rd, 2.0 KiB/s wr, 631 op/s 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:1684 (132): Start update from 127 to 130 (discovery-trial153). 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:2068 (132): Update complete (127 -> 130) (discovery-trial153). 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 127 to 130 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:35 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test12, no need to remove 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_29461d02-252c-42e1-8f87-67b48090adee 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (127 -> 130) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:36.101788] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:36.101827] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.098 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:1684 (105): Start update from 127 to 130 (discovery-trial183). 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:2068 (105): Update complete (127 -> 130) (discovery-trial183). 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 127 to 130 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:35 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test12, no need to remove 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_29461d02-252c-42e1-8f87-67b48090adee 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (127 -> 130) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:36.101401] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.099 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:36.101439] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.104 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 23 from nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:36.125 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:36.126 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode3 --nsid 24 2026-02-24T16:46:36.163 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:36Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:36.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:1684 (2): Start update from 127 to 130 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:36.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO grpc.py:4463 (2): Received request to delete namespace 22 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO grpc.py:3723 (2): Received request to remove namespace 22 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:1684 (126): Start update from 127 to 130 (discovery-trial191). 2026-02-24T16:46:36.341 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:35 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:35] INFO state.py:2068 (126): Update complete (127 -> 130) (discovery-trial191). 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test12, no need to remove 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_29461d02-252c-42e1-8f87-67b48090adee 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (127 -> 130) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:36.105794] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.342 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:36.105832] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.483 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63, client address: IPv4 10.20.193.52:54824 2026-02-24T16:46:36.737 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 24 from nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:36.759 INFO:tasks.workunit.client.1.trial052.stderr:+ for nsid in $NSIDs 2026-02-24T16:46:36.759 INFO:tasks.workunit.client.1.trial052.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 namespace del --subsystem nqn.2016-06.io.spdk:cnode3 --nsid 25 2026-02-24T16:46:36.799 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:36Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO state.py:1684 (132): Start update from 130 to 133 (discovery-trial153). 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO state.py:1684 (2): Start update from 130 to 133 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO state.py:2068 (132): Update complete (130 -> 133) (discovery-trial153). 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test13, no need to remove 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (130 -> 133) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:36.814 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:36] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:36.734508] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.815 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:36 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:36.734549] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO state.py:1684 (126): Start update from 130 to 133 (discovery-trial191). 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO state.py:2068 (126): Update complete (130 -> 133) (discovery-trial191). 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO state.py:1684 (2): Start update from 130 to 133 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test13, no need to remove 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (130 -> 133) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:36] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:36.738613] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.842 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:36 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:36.738644] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO state.py:1684 (2): Start update from 130 to 133 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO state.py:1684 (105): Start update from 130 to 133 (discovery-trial183). 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO state.py:2068 (105): Update complete (130 -> 133) (discovery-trial183). 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO grpc.py:4463 (2): Received request to delete namespace 23 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO grpc.py:3723 (2): Received request to remove namespace 23 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test13, no need to remove 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_8c20d236-b6aa-48fc-a6a4-f5b39a572d63 2026-02-24T16:46:36.984 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] INFO state.py:2068 (2): Update complete (130 -> 133) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:36.733819] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:36 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:36.733856] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:1684 (81): Start update from 130 to 133 (discovery-trial151). 2026-02-24T16:46:36.985 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:2068 (81): Update complete (130 -> 133) (discovery-trial151). 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:39786 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode3, client address: IPv4 10.20.193.52:39786 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:36.731854] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:36.731890] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:36.986 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:36 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:36] INFO grpc.py:1773 (2): Received request to delete bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96, client address: IPv4 10.20.193.52:39786 2026-02-24T16:46:37.322 INFO:tasks.workunit.client.1.trial052.stdout:Deleting namespace 25 from nqn.2016-06.io.spdk:cnode3: Successful 2026-02-24T16:46:37.343 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count_per_subsys=20 2026-02-24T16:46:37.343 INFO:tasks.workunit.client.1.trial052.stderr:+ assert_namespaces_count 20 2026-02-24T16:46:37.343 INFO:tasks.workunit.client.1.trial052.stderr:+ expected_count_per_subsys=20 2026-02-24T16:46:37.344 INFO:tasks.workunit.client.1.trial052.stderr:++ sudo podman run -it quay.io/ceph/nvmeof-cli:devel --server-address 10.20.193.151 --server-port 5500 --format json subsystem list 2026-02-24T16:46:37.344 INFO:tasks.workunit.client.1.trial052.stderr:++ grep namespace_count 2026-02-24T16:46:37.345 INFO:tasks.workunit.client.1.trial052.stderr:++ grep 20 2026-02-24T16:46:37.345 INFO:tasks.workunit.client.1.trial052.stderr:++ wc -l 2026-02-24T16:46:37.384 INFO:tasks.workunit.client.1.trial052.stderr:time="2026-02-24T16:46:37Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:1684 (105): Start update from 133 to 136 (discovery-trial183). 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:2068 (105): Update complete (133 -> 136) (discovery-trial183). 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 133 to 136 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test14, no need to remove 2026-02-24T16:46:37.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 2026-02-24T16:46:37.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (133 -> 136) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:37.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] WARNING state.py:640 (105): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:37.319369] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:37.485 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [2026-02-24 16:46:37.319405] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:1684 (81): Start update from 133 to 136 (discovery-trial151). 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:2068 (81): Update complete (133 -> 136) (discovery-trial151). 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 10.20.193.52:37608 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:603 (2): Succeeded to lock OMAP file (exclusive) after 1 retries 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode3, client address: IPv4 10.20.193.52:37608 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] WARNING state.py:640 (81): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:618 (2): No need to lock OMAP for read as we already have it locked for write 2026-02-24T16:46:37.486 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:37.317091] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:37.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [2026-02-24 16:46:37.317129] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:37.487 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111, client address: IPv4 10.20.193.52:37608 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 133 to 136 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:1684 (132): Start update from 133 to 136 (discovery-trial153). 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:2068 (132): Update complete (133 -> 136) (discovery-trial153). 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test14, no need to remove 2026-02-24T16:46:37.564 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 2026-02-24T16:46:37.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (133 -> 136) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:37.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] WARNING state.py:640 (132): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:37.319847] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:37.565 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [2026-02-24 16:46:37.319882] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:37.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 133 to 136 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:37.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 24 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:37.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 24 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:37.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:1684 (126): Start update from 133 to 136 (discovery-trial191). 2026-02-24T16:46:37.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:2068 (126): Update complete (133 -> 136) (discovery-trial191). 2026-02-24T16:46:37.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test14, no need to remove 2026-02-24T16:46:37.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_987771c0-d05d-4b2a-9f0b-eeca599a2c96 2026-02-24T16:46:37.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (133 -> 136) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:37.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] WARNING state.py:640 (2): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.593 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] WARNING state.py:640 (126): The OMAP file is locked, will try again in 0.5 seconds 2026-02-24T16:46:37.593 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:37.323906] bdev_rbd.c:2013:rbd_bdev_check_epoch: *NOTICE*: Failed to get metadata key = reservation_key 2026-02-24T16:46:37.593 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [2026-02-24 16:46:37.323941] bdev_rbd.c: 145:_rbd_update_callback: *NOTICE*: failed to notify reservation change. 2026-02-24T16:46:37.881 INFO:tasks.workunit.client.1.trial052.stderr:+ actual_count=3 2026-02-24T16:46:37.881 INFO:tasks.workunit.client.1.trial052.stderr:+ '[' 3 -ne 3 ']' 2026-02-24T16:46:37.881 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T16:46:37.882 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2026-02-24T16:46:37.968 INFO:tasks.workunit:Stopping ['nvmeof/basic_tests.sh', 'nvmeof/namespace_test.sh'] on client.1... 2026-02-24T16:46:37.968 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2026-02-24T16:46:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-mon[44017]: pgmap v207: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 2.0 KiB/s wr, 799 op/s 2026-02-24T16:46:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:1684 (132): Start update from 136 to 139 (discovery-trial153). 2026-02-24T16:46:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:2068 (132): Update complete (136 -> 139) (discovery-trial153). 2026-02-24T16:46:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 136 to 139 (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:38.064 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test15, no need to remove 2026-02-24T16:46:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 2026-02-24T16:46:38.065 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:37 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (136 -> 139) (gateway-client.nvmeof.nvmeof.b). 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 136 to 139 (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:1684 (126): Start update from 136 to 139 (discovery-trial191). 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:2068 (126): Update complete (136 -> 139) (discovery-trial191). 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test15, no need to remove 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 2026-02-24T16:46:38.091 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:37 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (136 -> 139) (gateway-client.nvmeof.nvmeof.d). 2026-02-24T16:46:38.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-mon[43553]: pgmap v207: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 2.0 KiB/s wr, 799 op/s 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:1684 (2): Start update from 136 to 139 (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:4463 (2): Received request to delete namespace 25 from nqn.2016-06.io.spdk:cnode3, I am sure, context: None 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:3723 (2): Received request to remove namespace 25 from nqn.2016-06.io.spdk:cnode3 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:1684 (105): Start update from 136 to 139 (discovery-trial183). 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:2068 (105): Update complete (136 -> 139) (discovery-trial183). 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO cephutils.py:599 (2): No metadata NVME_IMAGE_IDENTIFICATION for image mypool/test15, no need to remove 2026-02-24T16:46:38.234 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO grpc.py:1773 (2): Received request to delete bdev bdev_962a91c0-43ce-4a6d-8a89-ff002b365111 2026-02-24T16:46:38.235 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:37 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:37] INFO state.py:2068 (2): Update complete (136 -> 139) (gateway-client.nvmeof.nvmeof.c). 2026-02-24T16:46:38.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-mon[39453]: pgmap v207: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.0 MiB/s rd, 2.0 KiB/s wr, 799 op/s 2026-02-24T16:46:38.235 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:1684 (81): Start update from 136 to 139 (discovery-trial151). 2026-02-24T16:46:38.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO state.py:2068 (81): Update complete (136 -> 139) (discovery-trial151). 2026-02-24T16:46:38.236 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:37 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:37] INFO grpc.py:6951 (2): Received request to list all subsystems, context: , client address: IPv4 10.20.193.52:49072 2026-02-24T16:46:38.286 DEBUG:teuthology.parallel:result is None 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:38 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2028553184' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:38 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/324280714' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:38 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/872664675' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:38 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4050991215' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:38 trial151 ceph-mon[39453]: pgmap v208: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.5 KiB/s wr, 768 op/s 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:38 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2028553184' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:38 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/324280714' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:38 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/872664675' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:38 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4050991215' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:38 trial183 ceph-mon[43553]: pgmap v208: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.5 KiB/s wr, 768 op/s 2026-02-24T16:46:39.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:38 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2028553184' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:38 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/324280714' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:38 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/872664675' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:38 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4050991215' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:39.314 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:38 trial153 ceph-mon[44017]: pgmap v208: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.3 MiB/s rd, 3.5 KiB/s wr, 768 op/s 2026-02-24T16:46:41.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:40 trial151 ceph-mon[39453]: pgmap v209: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.0 KiB/s wr, 754 op/s 2026-02-24T16:46:41.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:40 trial183 ceph-mon[43553]: pgmap v209: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.0 KiB/s wr, 754 op/s 2026-02-24T16:46:41.313 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:40 trial153 ceph-mon[44017]: pgmap v209: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.0 KiB/s wr, 754 op/s 2026-02-24T16:46:43.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:43 trial183 ceph-mon[43553]: pgmap v210: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 736 op/s 2026-02-24T16:46:43.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:43 trial151 ceph-mon[39453]: pgmap v210: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 736 op/s 2026-02-24T16:46:44.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:43 trial153 ceph-mon[44017]: pgmap v210: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 1.2 MiB/s rd, 3.2 KiB/s wr, 736 op/s 2026-02-24T16:46:45.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:45 trial183 ceph-mon[43553]: pgmap v211: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1806731039' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2543772218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3942766794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:45 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3706401748' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:45 trial151 ceph-mon[39453]: pgmap v211: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1806731039' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2543772218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3942766794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:45.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:45 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3706401748' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:45 trial153 ceph-mon[44017]: pgmap v211: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1806731039' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2543772218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3942766794' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:45 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3706401748' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:47 trial151 ceph-mon[39453]: pgmap v212: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:47 trial183 ceph-mon[43553]: pgmap v212: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:47 trial153 ceph-mon[44017]: pgmap v212: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 705 KiB/s rd, 1.7 KiB/s wr, 437 op/s 2026-02-24T16:46:48.483 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:48] INFO prometheus.py:129 (2): Prometheus exporter running in http mode, listening on port 10008 2026-02-24T16:46:48.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:48] INFO grpc.py:7345 (2): Received request to get gateway's info 2026-02-24T16:46:48.484 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:46:48 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:46:48] INFO prometheus.py:183 (2): Stats for all bdevs will be provided 2026-02-24T16:46:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:48] INFO prometheus.py:129 (2): Prometheus exporter running in http mode, listening on port 10008 2026-02-24T16:46:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:48] INFO grpc.py:7345 (2): Received request to get gateway's info 2026-02-24T16:46:49.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:46:48 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:46:48] INFO prometheus.py:183 (2): Stats for all bdevs will be provided 2026-02-24T16:46:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:49 trial183 ceph-mon[43553]: pgmap v213: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 595 KiB/s rd, 1.7 KiB/s wr, 273 op/s 2026-02-24T16:46:49.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:49 trial151 ceph-mon[39453]: pgmap v213: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 595 KiB/s rd, 1.7 KiB/s wr, 273 op/s 2026-02-24T16:46:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:49 trial153 ceph-mon[44017]: pgmap v213: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 595 KiB/s rd, 1.7 KiB/s wr, 273 op/s 2026-02-24T16:46:50.483 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:50] INFO prometheus.py:129 (2): Prometheus exporter running in http mode, listening on port 10008 2026-02-24T16:46:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:50] INFO grpc.py:7345 (2): Received request to get gateway's info 2026-02-24T16:46:50.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:46:50 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:46:50] INFO prometheus.py:183 (2): Stats for all bdevs will be provided 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: pgmap v214: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 167 KiB/s rd, 255 B/s wr, 128 op/s 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:51 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:51] INFO prometheus.py:129 (2): Prometheus exporter running in http mode, listening on port 10008 2026-02-24T16:46:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:51] INFO grpc.py:7345 (2): Received request to get gateway's info 2026-02-24T16:46:51.841 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:46:51 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:46:51] INFO prometheus.py:183 (2): Stats for all bdevs will be provided 2026-02-24T16:46:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: pgmap v214: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 167 KiB/s rd, 255 B/s wr, 128 op/s 2026-02-24T16:46:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:46:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:51 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: pgmap v214: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 167 KiB/s rd, 255 B/s wr, 128 op/s 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:51.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:51 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/139517667' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/859861265' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3356386196' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:52 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1156507397' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/139517667' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:46:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:46:52.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:52.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:46:52.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/859861265' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3356386196' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:52.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:52 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1156507397' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/139517667' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/859861265' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3356386196' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:52 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1156507397' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:53.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:53 trial183 ceph-mon[43553]: pgmap v215: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 158 KiB/s rd, 255 B/s wr, 92 op/s 2026-02-24T16:46:53.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:53 trial151 ceph-mon[39453]: pgmap v215: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 158 KiB/s rd, 255 B/s wr, 92 op/s 2026-02-24T16:46:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:53 trial153 ceph-mon[44017]: pgmap v215: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 158 KiB/s rd, 255 B/s wr, 92 op/s 2026-02-24T16:46:55.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:55 trial151 ceph-mon[39453]: pgmap v216: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:55.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:55 trial183 ceph-mon[43553]: pgmap v216: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:56.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:55 trial153 ceph-mon[44017]: pgmap v216: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:57.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:57 trial151 ceph-mon[39453]: pgmap v217: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:57.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:57 trial183 ceph-mon[43553]: pgmap v217: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:58.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:57 trial153 ceph-mon[44017]: pgmap v217: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:46:59.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:59 trial183 ceph-mon[43553]: pgmap v218: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:46:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:59 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3554346574' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:59 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/606803544' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:59 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/888125721' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:46:59 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1414987899' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:59 trial151 ceph-mon[39453]: pgmap v218: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:46:59.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:59 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3554346574' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:59 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/606803544' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:59 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/888125721' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:46:59.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:46:59 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1414987899' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:00.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:59 trial153 ceph-mon[44017]: pgmap v218: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:59 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3554346574' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:59 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/606803544' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:59 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/888125721' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:46:59 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1414987899' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:01 trial151 ceph-mon[39453]: pgmap v219: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:01 trial183 ceph-mon[43553]: pgmap v219: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:02.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:01 trial153 ceph-mon[44017]: pgmap v219: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:03.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:03 trial151 ceph-mon[39453]: pgmap v220: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:03.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:03 trial183 ceph-mon[43553]: pgmap v220: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:04.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:03 trial153 ceph-mon[44017]: pgmap v220: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:05.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:05 trial151 ceph-mon[39453]: pgmap v221: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:05 trial183 ceph-mon[43553]: pgmap v221: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:06.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:05 trial153 ceph-mon[44017]: pgmap v221: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:06.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:06 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/511876927' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:06 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/407620237' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:06 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1961626382' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:06 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3746814145' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:06 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/511876927' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:06 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/407620237' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:06 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1961626382' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:06.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:06 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3746814145' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:07.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:06 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/511876927' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:07.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:06 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/407620237' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:07.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:06 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1961626382' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:07.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:06 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3746814145' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:07.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:07 trial151 ceph-mon[39453]: pgmap v222: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:07 trial183 ceph-mon[43553]: pgmap v222: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:08.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:07 trial153 ceph-mon[44017]: pgmap v222: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:09.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:09 trial151 ceph-mon[39453]: pgmap v223: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:09 trial183 ceph-mon[43553]: pgmap v223: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:09 trial153 ceph-mon[44017]: pgmap v223: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:11.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:11 trial151 ceph-mon[39453]: pgmap v224: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:11.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:11 trial183 ceph-mon[43553]: pgmap v224: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:12.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:11 trial153 ceph-mon[44017]: pgmap v224: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:13.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:13 trial151 ceph-mon[39453]: pgmap v225: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4087367907' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/237572323' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1772872694' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:13 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2635803686' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:13 trial183 ceph-mon[43553]: pgmap v225: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4087367907' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/237572323' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1772872694' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:13.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:13 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2635803686' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:13 trial153 ceph-mon[44017]: pgmap v225: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4087367907' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:14.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/237572323' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1772872694' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:13 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2635803686' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:15 trial151 ceph-mon[39453]: pgmap v226: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:15 trial183 ceph-mon[43553]: pgmap v226: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:15 trial153 ceph-mon[44017]: pgmap v226: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:17.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:17 trial151 ceph-mon[39453]: pgmap v227: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:17.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:17 trial183 ceph-mon[43553]: pgmap v227: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:17 trial153 ceph-mon[44017]: pgmap v227: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:19.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:19 trial151 ceph-mon[39453]: pgmap v228: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:19 trial183 ceph-mon[43553]: pgmap v228: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:20.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:19 trial153 ceph-mon[44017]: pgmap v228: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:20.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1571353758' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3366226991' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2122512664' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:20 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/490086485' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1571353758' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3366226991' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2122512664' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:20.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:20 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/490086485' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1571353758' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3366226991' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2122512664' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:21.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:20 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/490086485' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:21.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:21 trial151 ceph-mon[39453]: pgmap v229: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:21.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:21 trial183 ceph-mon[43553]: pgmap v229: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:21 trial153 ceph-mon[44017]: pgmap v229: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:23 trial151 ceph-mon[39453]: pgmap v230: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:23.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:23 trial183 ceph-mon[43553]: pgmap v230: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:24.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:23 trial153 ceph-mon[44017]: pgmap v230: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:25.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:25 trial151 ceph-mon[39453]: pgmap v231: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:25.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:25 trial183 ceph-mon[43553]: pgmap v231: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:25 trial153 ceph-mon[44017]: pgmap v231: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:27.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:27 trial151 ceph-mon[39453]: pgmap v232: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1819667309' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/948476813' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4174124914' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:27 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/32987889' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:27 trial183 ceph-mon[43553]: pgmap v232: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:27.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1819667309' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/948476813' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4174124914' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:27.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:27 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/32987889' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:27 trial153 ceph-mon[44017]: pgmap v232: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1819667309' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/948476813' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:28.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4174124914' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:28.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:27 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/32987889' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:29 trial151 ceph-mon[39453]: pgmap v233: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:29.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:29 trial183 ceph-mon[43553]: pgmap v233: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:29 trial153 ceph-mon[44017]: pgmap v233: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:31.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:31 trial151 ceph-mon[39453]: pgmap v234: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:31.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:31 trial183 ceph-mon[43553]: pgmap v234: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:32.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:31 trial153 ceph-mon[44017]: pgmap v234: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:33.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:33 trial151 ceph-mon[39453]: pgmap v235: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:33.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:33 trial183 ceph-mon[43553]: pgmap v235: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:33 trial153 ceph-mon[44017]: pgmap v235: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:34.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2527317440' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4105527439' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1651907969' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:34 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/780586010' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2527317440' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4105527439' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1651907969' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:34.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:34 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/780586010' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2527317440' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4105527439' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1651907969' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:35.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:34 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/780586010' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:35.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:35 trial151 ceph-mon[39453]: pgmap v236: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:35.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:35 trial183 ceph-mon[43553]: pgmap v236: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:36.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:35 trial153 ceph-mon[44017]: pgmap v236: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:37.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:37 trial151 ceph-mon[39453]: pgmap v237: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:37.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:37 trial183 ceph-mon[43553]: pgmap v237: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:37 trial153 ceph-mon[44017]: pgmap v237: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:39.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:39 trial151 ceph-mon[39453]: pgmap v238: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:39 trial183 ceph-mon[43553]: pgmap v238: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:40.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:39 trial153 ceph-mon[44017]: pgmap v238: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:41.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:41 trial151 ceph-mon[39453]: pgmap v239: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2113629086' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/177248014' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1276714466' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:41 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1360273350' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:41 trial183 ceph-mon[43553]: pgmap v239: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:41.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2113629086' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/177248014' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1276714466' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:41.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:41 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1360273350' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:41 trial153 ceph-mon[44017]: pgmap v239: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2113629086' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/177248014' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1276714466' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:41 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1360273350' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:43.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:43 trial151 ceph-mon[39453]: pgmap v240: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:43.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:43 trial183 ceph-mon[43553]: pgmap v240: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:43 trial153 ceph-mon[44017]: pgmap v240: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:45.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:45 trial151 ceph-mon[39453]: pgmap v241: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:45 trial183 ceph-mon[43553]: pgmap v241: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:45 trial153 ceph-mon[44017]: pgmap v241: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:47 trial151 ceph-mon[39453]: pgmap v242: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:47 trial183 ceph-mon[43553]: pgmap v242: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:47 trial153 ceph-mon[44017]: pgmap v242: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:48.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1021337921' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/165872812' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3603541340' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:48 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3672613212' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1021337921' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/165872812' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3603541340' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:48.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:48 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3672613212' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:49.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1021337921' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:49.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/165872812' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:49.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3603541340' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:49.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:48 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3672613212' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:49.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:49 trial151 ceph-mon[39453]: pgmap v243: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:49 trial183 ceph-mon[43553]: pgmap v243: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:49 trial153 ceph-mon[44017]: pgmap v243: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:51.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:51 trial151 ceph-mon[39453]: pgmap v244: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:51.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:51 trial183 ceph-mon[43553]: pgmap v244: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:51 trial153 ceph-mon[44017]: pgmap v244: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:52.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:52 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:47:52.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:52.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:52 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:52 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:53 trial151 ceph-mon[39453]: pgmap v245: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:53 trial183 ceph-mon[43553]: pgmap v245: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:53.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:47:53.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:47:53.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:53.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:47:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:53 trial153 ceph-mon[44017]: pgmap v245: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:47:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:47:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:47:54.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:47:55.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:55 trial151 ceph-mon[39453]: pgmap v246: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/381287324' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1365192269' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/905483430' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:55 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/608938885' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:55 trial183 ceph-mon[43553]: pgmap v246: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:55.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/381287324' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1365192269' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/905483430' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:55.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:55 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/608938885' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:55 trial153 ceph-mon[44017]: pgmap v246: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/381287324' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1365192269' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/905483430' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:55 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/608938885' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:47:57.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:57 trial151 ceph-mon[39453]: pgmap v247: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:57.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:57 trial183 ceph-mon[43553]: pgmap v247: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:57 trial153 ceph-mon[44017]: pgmap v247: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:47:59.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:47:59 trial151 ceph-mon[39453]: pgmap v248: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:47:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:47:59 trial183 ceph-mon[43553]: pgmap v248: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:47:59 trial153 ceph-mon[44017]: pgmap v248: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:01 trial151 ceph-mon[39453]: pgmap v249: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:01.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:01 trial183 ceph-mon[43553]: pgmap v249: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:01 trial153 ceph-mon[44017]: pgmap v249: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:02.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/385582189' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/251796006' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4023065273' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:02 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1007984736' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/385582189' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/251796006' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4023065273' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:02.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:02 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1007984736' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:03.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/385582189' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/251796006' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4023065273' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:02 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1007984736' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:03.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:03 trial151 ceph-mon[39453]: pgmap v250: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:03.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:03 trial183 ceph-mon[43553]: pgmap v250: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:04.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:03 trial153 ceph-mon[44017]: pgmap v250: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:05.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:05 trial151 ceph-mon[39453]: pgmap v251: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:05 trial183 ceph-mon[43553]: pgmap v251: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:05 trial153 ceph-mon[44017]: pgmap v251: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:07.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:07 trial151 ceph-mon[39453]: pgmap v252: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:07 trial183 ceph-mon[43553]: pgmap v252: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:07 trial153 ceph-mon[44017]: pgmap v252: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:09.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:09 trial151 ceph-mon[39453]: pgmap v253: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/645566529' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3753701631' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1680303780' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:09 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/732951568' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:09 trial183 ceph-mon[43553]: pgmap v253: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:09.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/645566529' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3753701631' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1680303780' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:09.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:09 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/732951568' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:09 trial153 ceph-mon[44017]: pgmap v253: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/645566529' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3753701631' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1680303780' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:09 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/732951568' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:11.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:11 trial151 ceph-mon[39453]: pgmap v254: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:11.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:11 trial183 ceph-mon[43553]: pgmap v254: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:12.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:11 trial153 ceph-mon[44017]: pgmap v254: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:13.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:13 trial151 ceph-mon[39453]: pgmap v255: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:13.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:13 trial183 ceph-mon[43553]: pgmap v255: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:13 trial153 ceph-mon[44017]: pgmap v255: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:15 trial151 ceph-mon[39453]: pgmap v256: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:15 trial183 ceph-mon[43553]: pgmap v256: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:15 trial153 ceph-mon[44017]: pgmap v256: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1227327674' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/397946087' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1694961159' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:16 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1893542378' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1227327674' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/397946087' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1694961159' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:16.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:16 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1893542378' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1227327674' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/397946087' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1694961159' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:17.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:16 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1893542378' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:17.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:17 trial151 ceph-mon[39453]: pgmap v257: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:17.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:17 trial183 ceph-mon[43553]: pgmap v257: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:17 trial153 ceph-mon[44017]: pgmap v257: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:19.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:19 trial151 ceph-mon[39453]: pgmap v258: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:19.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:19 trial183 ceph-mon[43553]: pgmap v258: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:20.063 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:19 trial153 ceph-mon[44017]: pgmap v258: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:21.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:21 trial151 ceph-mon[39453]: pgmap v259: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:21.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:21 trial183 ceph-mon[43553]: pgmap v259: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:21 trial153 ceph-mon[44017]: pgmap v259: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:23 trial151 ceph-mon[39453]: pgmap v260: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2843855533' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3095242874' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2039257578' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:23 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/172334474' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:23 trial183 ceph-mon[43553]: pgmap v260: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:23.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2843855533' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3095242874' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2039257578' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:23.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:23 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/172334474' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:23 trial153 ceph-mon[44017]: pgmap v260: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2843855533' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3095242874' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2039257578' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:23 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/172334474' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:25 trial153 ceph-mon[44017]: pgmap v261: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:26.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:25 trial151 ceph-mon[39453]: pgmap v261: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:26.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:25 trial183 ceph-mon[43553]: pgmap v261: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:27 trial153 ceph-mon[44017]: pgmap v262: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:28.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:27 trial151 ceph-mon[39453]: pgmap v262: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:28.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:27 trial183 ceph-mon[43553]: pgmap v262: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:30.035 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:29 trial151 ceph-mon[39453]: pgmap v263: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:29 trial153 ceph-mon[44017]: pgmap v263: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:30.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:29 trial183 ceph-mon[43553]: pgmap v263: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2430450369' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/243591804' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3134376804' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:30 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1851273772' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.134 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2430450369' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.135 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/243591804' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.135 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3134376804' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.135 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:30 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1851273772' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2430450369' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/243591804' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3134376804' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:31.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:30 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1851273772' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:32.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:31 trial153 ceph-mon[44017]: pgmap v264: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:32.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:31 trial151 ceph-mon[39453]: pgmap v264: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:32.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:31 trial183 ceph-mon[43553]: pgmap v264: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:33 trial153 ceph-mon[44017]: pgmap v265: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:34.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:33 trial151 ceph-mon[39453]: pgmap v265: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:34.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:33 trial183 ceph-mon[43553]: pgmap v265: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:36.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:35 trial153 ceph-mon[44017]: pgmap v266: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:36.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:35 trial151 ceph-mon[39453]: pgmap v266: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:36.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:35 trial183 ceph-mon[43553]: pgmap v266: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:37 trial153 ceph-mon[44017]: pgmap v267: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/924954641' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/22419315' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/517391673' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:37 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2323322285' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:37 trial151 ceph-mon[39453]: pgmap v267: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/924954641' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/22419315' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/517391673' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:37 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2323322285' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:37 trial183 ceph-mon[43553]: pgmap v267: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:38.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/924954641' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/22419315' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/517391673' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:38.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:37 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2323322285' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:39 trial153 ceph-mon[44017]: pgmap v268: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:40.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:39 trial151 ceph-mon[39453]: pgmap v268: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:40.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:39 trial183 ceph-mon[43553]: pgmap v268: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:41 trial153 ceph-mon[44017]: pgmap v269: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:42.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:41 trial151 ceph-mon[39453]: pgmap v269: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:42.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:41 trial183 ceph-mon[43553]: pgmap v269: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:43 trial153 ceph-mon[44017]: pgmap v270: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:44.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:43 trial151 ceph-mon[39453]: pgmap v270: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:44.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:43 trial183 ceph-mon[43553]: pgmap v270: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/870405831' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1943482462' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2595047402' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:44 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2075397284' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/870405831' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1943482462' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2595047402' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:44 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2075397284' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/870405831' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1943482462' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2595047402' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:45.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:44 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2075397284' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:45 trial153 ceph-mon[44017]: pgmap v271: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:46.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:45 trial151 ceph-mon[39453]: pgmap v271: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:46.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:45 trial183 ceph-mon[43553]: pgmap v271: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:47 trial153 ceph-mon[44017]: pgmap v272: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:48.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:47 trial151 ceph-mon[39453]: pgmap v272: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:48.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:47 trial183 ceph-mon[43553]: pgmap v272: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:49 trial153 ceph-mon[44017]: pgmap v273: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:50.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:49 trial151 ceph-mon[39453]: pgmap v273: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:50.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:49 trial183 ceph-mon[43553]: pgmap v273: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:51 trial153 ceph-mon[44017]: pgmap v274: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1877132134' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4272890022' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/64316066' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:51 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4198955638' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:51 trial151 ceph-mon[39453]: pgmap v274: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:52.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1877132134' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4272890022' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/64316066' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:51 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4198955638' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:51 trial183 ceph-mon[43553]: pgmap v274: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:52.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1877132134' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4272890022' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/64316066' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:51 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4198955638' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:52.584 INFO:tasks.workunit.client.0.trial051.stderr:Note: switching to '618c318a7a0b287ab5aeff207b7bf1d73ef45044'. 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr:state without impacting any branches by switching back to a branch. 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr: git switch -c 2026-02-24T16:48:52.585 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr:Or undo this operation with: 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr: git switch - 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:52.586 INFO:tasks.workunit.client.0.trial051.stderr:HEAD is now at 618c318a7a0 Merge branch 'stretched-cluster' of https://github.com/leonidc/ceph into wip-yuri8-testing-2026-02-23-2128 2026-02-24T16:48:52.592 DEBUG:teuthology.orchestra.run.trial051:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-24T16:48:52.653 INFO:tasks.workunit.client.0.trial051.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-24T16:48:52.655 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T16:48:52.655 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-24T16:48:52.687 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-24T16:48:52.717 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-24T16:48:52.742 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T16:48:52.744 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T16:48:52.744 INFO:tasks.workunit.client.0.trial051.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-24T16:48:52.769 INFO:tasks.workunit.client.0.trial051.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T16:48:52.772 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-24T16:48:52.773 DEBUG:teuthology.orchestra.run.trial051:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-24T16:48:52.830 INFO:tasks.workunit:Running workunits matching nvmeof/fio_test.sh on client.0... 2026-02-24T16:48:52.833 INFO:tasks.workunit:Running workunit nvmeof/fio_test.sh... 2026-02-24T16:48:52.833 DEBUG:teuthology.orchestra.run.trial051:workunit test nvmeof/fio_test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 IOSTAT_INTERVAL=10 NEW_NAMESPACES_COUNT=5 RBD_POOL=mypool RUNTIME=120 timeout 30m /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/fio_test.sh --rbd_iostat 2026-02-24T16:48:52.888 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo yum -y install fio 2026-02-24T16:48:53.206 INFO:tasks.workunit.client.0.trial051.stdout:Last metadata expiration check: 0:11:02 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:48:53.290 INFO:tasks.workunit.client.0.trial051.stdout:Dependencies resolved. 2026-02-24T16:48:53.291 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:53.291 INFO:tasks.workunit.client.0.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:48:53.291 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout:Installing: 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio x86_64 3.35-1.el9 appstream 6.2 M 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout:Installing weak dependencies: 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-http x86_64 3.35-1.el9 appstream 18 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-libaio x86_64 3.35-1.el9 appstream 16 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-nbd x86_64 3.35-1.el9 appstream 15 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-rados x86_64 3.35-1.el9 appstream 16 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-rbd x86_64 3.35-1.el9 appstream 16 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-rdma x86_64 3.35-1.el9 appstream 21 k 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout:Transaction Summary 2026-02-24T16:48:53.292 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:53.293 INFO:tasks.workunit.client.0.trial051.stdout:Install 7 Packages 2026-02-24T16:48:53.293 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:53.293 INFO:tasks.workunit.client.0.trial051.stdout:Total download size: 6.3 M 2026-02-24T16:48:53.293 INFO:tasks.workunit.client.0.trial051.stdout:Installed size: 8.7 M 2026-02-24T16:48:53.293 INFO:tasks.workunit.client.0.trial051.stdout:Downloading Packages: 2026-02-24T16:48:53.697 INFO:tasks.workunit.client.0.trial051.stdout:(1/7): fio-engine-libaio-3.35-1.el9.x86_64.rpm 366 kB/s | 16 kB 00:00 2026-02-24T16:48:53.697 INFO:tasks.workunit.client.0.trial051.stdout:(2/7): fio-engine-http-3.35-1.el9.x86_64.rpm 419 kB/s | 18 kB 00:00 2026-02-24T16:48:53.718 INFO:tasks.workunit.client.0.trial051.stdout:(3/7): fio-engine-nbd-3.35-1.el9.x86_64.rpm 680 kB/s | 15 kB 00:00 2026-02-24T16:48:53.719 INFO:tasks.workunit.client.0.trial051.stdout:(4/7): fio-engine-rados-3.35-1.el9.x86_64.rpm 732 kB/s | 16 kB 00:00 2026-02-24T16:48:53.740 INFO:tasks.workunit.client.0.trial051.stdout:(5/7): fio-engine-rbd-3.35-1.el9.x86_64.rpm 758 kB/s | 16 kB 00:00 2026-02-24T16:48:53.740 INFO:tasks.workunit.client.0.trial051.stdout:(6/7): fio-engine-rdma-3.35-1.el9.x86_64.rpm 967 kB/s | 21 kB 00:00 2026-02-24T16:48:53.993 INFO:tasks.workunit.client.0.trial051.stdout:(7/7): fio-3.35-1.el9.x86_64.rpm 18 MB/s | 6.2 MB 00:00 2026-02-24T16:48:53.993 INFO:tasks.workunit.client.0.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:48:53.993 INFO:tasks.workunit.client.0.trial051.stdout:Total 9.0 MB/s | 6.3 MB 00:00 2026-02-24T16:48:54.062 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction check 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: pgmap v275: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:53 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:48:54.069 INFO:tasks.workunit.client.0.trial051.stdout:Transaction check succeeded. 2026-02-24T16:48:54.069 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction test 2026-02-24T16:48:54.128 INFO:tasks.workunit.client.0.trial051.stdout:Transaction test succeeded. 2026-02-24T16:48:54.128 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction 2026-02-24T16:48:54.205 INFO:tasks.workunit.client.0.trial051.stdout: Preparing : 1/1 2026-02-24T16:48:54.207 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-http-3.35-1.el9.x86_64 1/7 2026-02-24T16:48:54.208 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-libaio-3.35-1.el9.x86_64 2/7 2026-02-24T16:48:54.210 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-nbd-3.35-1.el9.x86_64 3/7 2026-02-24T16:48:54.211 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-rados-3.35-1.el9.x86_64 4/7 2026-02-24T16:48:54.212 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-rbd-3.35-1.el9.x86_64 5/7 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: pgmap v275: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:53 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: pgmap v275: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:48:54.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:53 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:48:54.268 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-engine-rdma-3.35-1.el9.x86_64 6/7 2026-02-24T16:48:54.287 INFO:tasks.workunit.client.0.trial051.stdout: Installing : fio-3.35-1.el9.x86_64 7/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: fio-3.35-1.el9.x86_64 7/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-3.35-1.el9.x86_64 1/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-http-3.35-1.el9.x86_64 2/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-libaio-3.35-1.el9.x86_64 3/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-nbd-3.35-1.el9.x86_64 4/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-rados-3.35-1.el9.x86_64 5/7 2026-02-24T16:48:54.557 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-rbd-3.35-1.el9.x86_64 6/7 2026-02-24T16:48:54.621 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : fio-engine-rdma-3.35-1.el9.x86_64 7/7 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout:Installed: 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: fio-3.35-1.el9.x86_64 fio-engine-http-3.35-1.el9.x86_64 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-libaio-3.35-1.el9.x86_64 fio-engine-nbd-3.35-1.el9.x86_64 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-rados-3.35-1.el9.x86_64 fio-engine-rbd-3.35-1.el9.x86_64 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: fio-engine-rdma-3.35-1.el9.x86_64 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:54.622 INFO:tasks.workunit.client.0.trial051.stdout:Complete! 2026-02-24T16:48:54.650 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo yum -y install sysstat 2026-02-24T16:48:54.951 INFO:tasks.workunit.client.0.trial051.stdout:Last metadata expiration check: 0:11:03 ago on Tue 24 Feb 2026 04:37:51 PM UTC. 2026-02-24T16:48:55.015 INFO:tasks.workunit.client.0.trial051.stdout:Package sysstat-12.5.4-9.el9.x86_64 is already installed. 2026-02-24T16:48:55.033 INFO:tasks.workunit.client.0.trial051.stdout:Dependencies resolved. 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout: Package Architecture Version Repository Size 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:Upgrading: 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout: sysstat x86_64 12.5.4-11.el9 appstream 480 k 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:Transaction Summary 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:================================================================================ 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:Upgrade 1 Package 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:Total download size: 480 k 2026-02-24T16:48:55.034 INFO:tasks.workunit.client.0.trial051.stdout:Downloading Packages: 2026-02-24T16:48:55.431 INFO:tasks.workunit.client.0.trial051.stdout:sysstat-12.5.4-11.el9.x86_64.rpm 5.1 MB/s | 480 kB 00:00 2026-02-24T16:48:55.432 INFO:tasks.workunit.client.0.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:48:55.432 INFO:tasks.workunit.client.0.trial051.stdout:Total 1.2 MB/s | 480 kB 00:00 2026-02-24T16:48:55.443 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction check 2026-02-24T16:48:55.448 INFO:tasks.workunit.client.0.trial051.stdout:Transaction check succeeded. 2026-02-24T16:48:55.448 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction test 2026-02-24T16:48:55.494 INFO:tasks.workunit.client.0.trial051.stdout:Transaction test succeeded. 2026-02-24T16:48:55.494 INFO:tasks.workunit.client.0.trial051.stdout:Running transaction 2026-02-24T16:48:55.574 INFO:tasks.workunit.client.0.trial051.stdout: Preparing : 1/1 2026-02-24T16:48:55.576 INFO:tasks.workunit.client.0.trial051.stdout: Upgrading : sysstat-12.5.4-11.el9.x86_64 1/2 2026-02-24T16:48:55.582 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: sysstat-12.5.4-11.el9.x86_64 1/2 2026-02-24T16:48:55.588 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: sysstat-12.5.4-9.el9.x86_64 2/2 2026-02-24T16:48:55.588 INFO:tasks.workunit.client.0.trial051.stdout: Cleanup : sysstat-12.5.4-9.el9.x86_64 2/2 2026-02-24T16:48:55.976 INFO:tasks.workunit.client.0.trial051.stdout: Running scriptlet: sysstat-12.5.4-9.el9.x86_64 2/2 2026-02-24T16:48:55.976 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : sysstat-12.5.4-11.el9.x86_64 1/2 2026-02-24T16:48:56.039 INFO:tasks.workunit.client.0.trial051.stdout: Verifying : sysstat-12.5.4-9.el9.x86_64 2/2 2026-02-24T16:48:56.040 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.040 INFO:tasks.workunit.client.0.trial051.stdout:Upgraded: 2026-02-24T16:48:56.040 INFO:tasks.workunit.client.0.trial051.stdout: sysstat-12.5.4-11.el9.x86_64 2026-02-24T16:48:56.040 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.040 INFO:tasks.workunit.client.0.trial051.stdout:Complete! 2026-02-24T16:48:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:55 trial153 ceph-mon[44017]: pgmap v276: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:56.067 INFO:tasks.workunit.client.0.trial051.stderr:+ namespace_range_start= 2026-02-24T16:48:56.067 INFO:tasks.workunit.client.0.trial051.stderr:+ namespace_range_end= 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ random_devices_count= 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ rbd_iostat=false 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ [[ 1 -gt 0 ]] 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ case "$1" in 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ rbd_iostat=true 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ shift 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:+ [[ 0 -gt 0 ]] 2026-02-24T16:48:56.068 INFO:tasks.workunit.client.0.trial051.stderr:++ mktemp -t nvmeof-fio-XXXX 2026-02-24T16:48:56.069 INFO:tasks.workunit.client.0.trial051.stderr:+ fio_file=/tmp/nvmeof-fio-W4pN 2026-02-24T16:48:56.069 INFO:tasks.workunit.client.0.trial051.stderr:++ sudo nvme list --output-format=json 2026-02-24T16:48:56.069 INFO:tasks.workunit.client.0.trial051.stderr:++ jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == "Ceph bdev Controller") | .DevicePath' 2026-02-24T16:48:56.101 INFO:tasks.workunit.client.0.trial051.stderr:+ all_drives_list='/dev/nvme8n1 2026-02-24T16:48:56.101 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n1 2026-02-24T16:48:56.101 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n1 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n2 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n2 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n2 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n3 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n3 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n3 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n4 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n4 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n4 2026-02-24T16:48:56.102 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n5 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n5 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n5 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n6 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n6 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n6 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n7 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n7 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n7 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n8 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n8 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n8 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n9 2026-02-24T16:48:56.103 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n9 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n9 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n10 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n10 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n10 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n11 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n11 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n11 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n12 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n12 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n12 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n13 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n13 2026-02-24T16:48:56.104 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n13 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n14 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n14 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n14 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n15 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n15 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n15 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n16 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n16 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n16 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n17 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n17 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n17 2026-02-24T16:48:56.105 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n18 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n18 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n18 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n19 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n19 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n19 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n20 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n20 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n20' 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' '' ']' 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' '' ']' 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' '' ']' 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:+ selected_drives='/dev/nvme8n1 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n1 2026-02-24T16:48:56.106 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n1 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n2 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n2 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n2 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n3 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n3 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n3 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n4 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n4 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n4 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n5 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n5 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n5 2026-02-24T16:48:56.107 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n6 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n6 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n6 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n7 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n7 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n7 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n8 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n8 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n8 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n9 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n9 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n9 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n10 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n10 2026-02-24T16:48:56.108 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n10 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n11 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n11 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n11 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n12 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n12 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n12 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n13 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n13 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n13 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n14 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n14 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n14 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n15 2026-02-24T16:48:56.109 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n15 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n15 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n16 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n16 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n16 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n17 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n17 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n17 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n18 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n18 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n18 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n19 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n19 2026-02-24T16:48:56.110 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n19 2026-02-24T16:48:56.111 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme8n20 2026-02-24T16:48:56.111 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme12n20 2026-02-24T16:48:56.111 INFO:tasks.workunit.client.0.trial051.stderr:/dev/nvme4n20' 2026-02-24T16:48:56.111 INFO:tasks.workunit.client.0.trial051.stderr:+ RUNTIME=120 2026-02-24T16:48:56.111 INFO:tasks.workunit.client.0.trial051.stderr:+ cat 2026-02-24T16:48:56.128 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.128 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n1]' 2026-02-24T16:48:56.128 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n1 2026-02-24T16:48:56.128 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n1]' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n1 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n1]' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n1 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n2]' 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n2 2026-02-24T16:48:56.129 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n2]' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n2 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n2]' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n2 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n3]' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n3 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.130 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n3]' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n3 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n3]' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n3 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n4]' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n4 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.131 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n4]' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n4 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n4]' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n4 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n5]' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n5 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.132 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n5]' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n5 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n5]' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n5 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n6]' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n6 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n6]' 2026-02-24T16:48:56.133 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n6 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n6]' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n6 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n7]' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n7 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n7]' 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n7 2026-02-24T16:48:56.134 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n7]' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n7 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n8]' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n8 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n8]' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n8 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.135 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n8]' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n8 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n9]' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n9 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n9]' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n9 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n9]' 2026-02-24T16:48:56.136 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n9 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n10]' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n10 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n10]' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n10 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n10]' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n10 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.137 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n11]' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n11 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n11]' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n11 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n11]' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n11 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n12]' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n12 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.138 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n12]' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n12 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n12]' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n12 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n13]' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n13 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n13]' 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n13 2026-02-24T16:48:56.139 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n13]' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n13 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n14]' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n14 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n14]' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n14 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n14]' 2026-02-24T16:48:56.140 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n14 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n15]' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n15 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n15]' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n15 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n15]' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n15 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n16]' 2026-02-24T16:48:56.141 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n16 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n16]' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n16 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n16]' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n16 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n17]' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n17 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n17]' 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n17 2026-02-24T16:48:56.142 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n17]' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n17 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n18]' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n18 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n18]' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n18 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n18]' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n18 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.143 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n19]' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n19 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n19]' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n19 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n19]' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n19 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme8n20]' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme8n20 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme12n20]' 2026-02-24T16:48:56.144 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme12n20 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ for i in $selected_drives 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[job-/dev/nvme4n20]' 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ echo filename=/dev/nvme4n20 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '' 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stderr:+ cat /tmp/nvmeof-fio-W4pN 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stdout:[global] 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stdout:ioengine=sync 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stdout:bsrange=4k-64k 2026-02-24T16:48:56.145 INFO:tasks.workunit.client.0.trial051.stdout:numjobs=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:size=1G 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:time_based=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:runtime=120 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:rw=randrw 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:verify=md5 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:verify_fatal=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:do_verify=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:serialize_overlap=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:group_reporting 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:direct=1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n1] 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n1] 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n1 2026-02-24T16:48:56.146 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n1] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n1 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n2] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n2 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n2] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n2 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n2] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n2 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n3] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n3 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n3] 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n3 2026-02-24T16:48:56.147 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n3] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n3 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n4] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n4 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n4] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n4 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n4] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n4 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n5] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n5 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n5] 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n5 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.148 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n5] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n5 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n6] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n6 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n6] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n6 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n6] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n6 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n7] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n7 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n7] 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n7 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.149 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n7] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n7 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n8] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n8 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n8] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n8 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n8] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n8 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n9] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n9 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n9] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n9 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n9] 2026-02-24T16:48:56.150 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n9 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n10] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n10 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n10] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n10 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n10] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n10 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n11] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n11 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n11] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n11 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n11] 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n11 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.151 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n12] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n12 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n12] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n12 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n12] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n12 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n13] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n13 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n13] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n13 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n13] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n13 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n14] 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n14 2026-02-24T16:48:56.152 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n14] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n14 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n14] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n14 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n15] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n15 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n15] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n15 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n15] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n15 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n16] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n16 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n16] 2026-02-24T16:48:56.153 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n16 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n16] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n16 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n17] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n17 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n17] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n17 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n17] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n17 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n18] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n18 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n18] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n18 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n18] 2026-02-24T16:48:56.154 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n18 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n19] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n19 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n19] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n19 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n19] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n19 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme8n20] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme8n20 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme12n20] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme12n20 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[job-/dev/nvme4n20] 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:filename=/dev/nvme4n20 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.155 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof.fio] starting fio test... 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof.fio] starting fio test...' 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' -n 10 ']' 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ iostat_count=12 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' true = true ']' 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ iterations=24 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ iostat -d -p /dev/nvme8n1 /dev/nvme12n1 /dev/nvme4n1 /dev/nvme8n2 /dev/nvme12n2 /dev/nvme4n2 /dev/nvme8n3 /dev/nvme12n3 /dev/nvme4n3 /dev/nvme8n4 /dev/nvme12n4 /dev/nvme4n4 /dev/nvme8n5 /dev/nvme12n5 /dev/nvme4n5 /dev/nvme8n6 /dev/nvme12n6 /dev/nvme4n6 /dev/nvme8n7 /dev/nvme12n7 /dev/nvme4n7 /dev/nvme8n8 /dev/nvme12n8 /dev/nvme4n8 /dev/nvme8n9 /dev/nvme12n9 /dev/nvme4n9 /dev/nvme8n10 /dev/nvme12n10 /dev/nvme4n10 /dev/nvme8n11 /dev/nvme12n11 /dev/nvme4n11 /dev/nvme8n12 /dev/nvme12n12 /dev/nvme4n12 /dev/nvme8n13 /dev/nvme12n13 /dev/nvme4n13 /dev/nvme8n14 /dev/nvme12n14 /dev/nvme4n14 /dev/nvme8n15 /dev/nvme12n15 /dev/nvme4n15 /dev/nvme8n16 /dev/nvme12n16 /dev/nvme4n16 /dev/nvme8n17 /dev/nvme12n17 /dev/nvme4n17 /dev/nvme8n18 /dev/nvme12n18 /dev/nvme4n18 /dev/nvme8n19 /dev/nvme12n19 /dev/nvme4n19 /dev/nvme8n20 /dev/nvme12n20 /dev/nvme4n20 10 12 -h 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ fio --showcmd /tmp/nvmeof-fio-W4pN 2026-02-24T16:48:56.156 INFO:tasks.workunit.client.0.trial051.stderr:+ timeout 20 rbd perf image iostat mypool --iterations 24 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout:Linux 5.14.0-686.el9.x86_64 (trial051) 02/24/2026 _x86_64_ (24 CPU) 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n1 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n10 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n11 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n12 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n13 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n14 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n15 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n16 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n17 2026-02-24T16:48:56.162 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n18 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n19 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n2 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n20 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n3 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n4 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n5 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n6 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n7 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n8 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme12n9 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n1 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n10 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n11 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n12 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n13 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n14 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n15 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n16 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n17 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n18 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n19 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n2 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n20 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n3 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n4 2026-02-24T16:48:56.163 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n5 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n6 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n7 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n8 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme4n9 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n1 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n10 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n11 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n12 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n13 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n14 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n15 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n16 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n17 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n18 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n19 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n2 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n20 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n3 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n4 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n5 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n6 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n7 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n8 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 0.05 1.1k 0.0k 0.0k 1.0M 0.0k 0.0k nvme8n9 2026-02-24T16:48:56.164 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:48:56.165 INFO:tasks.workunit.client.0.trial051.stderr:rbd: waiting for initial image stats 2026-02-24T16:48:56.165 INFO:tasks.workunit.client.0.trial051.stderr: 2026-02-24T16:48:56.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:55 trial151 ceph-mon[39453]: pgmap v276: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:56.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:55 trial183 ceph-mon[43553]: pgmap v276: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 0 B/s wr, 7 op/s 2026-02-24T16:48:56.245 INFO:tasks.workunit.client.0.trial051.stdout:fio --ioengine=sync --bsrange=4k-64k --numjobs=1 --size=1G --time_based=1 --runtime=120 --rw=randrw --verify=md5 --verify_fatal=1 --do_verify=1 --serialize_overlap=1 --group_reporting --direct=1 --name=job-/dev/nvme8n1 --filename=/dev/nvme8n1 --name=job-/dev/nvme12n1 --filename=/dev/nvme12n1 --name=job-/dev/nvme4n1 --filename=/dev/nvme4n1 --name=job-/dev/nvme8n2 --filename=/dev/nvme8n2 --name=job-/dev/nvme12n2 --filename=/dev/nvme12n2 --name=job-/dev/nvme4n2 --filename=/dev/nvme4n2 --name=job-/dev/nvme8n3 --filename=/dev/nvme8n3 --name=job-/dev/nvme12n3 --filename=/dev/nvme12n3 --name=job-/dev/nvme4n3 --filename=/dev/nvme4n3 --name=job-/dev/nvme8n4 --filename=/dev/nvme8n4 --name=job-/dev/nvme12n4 --filename=/dev/nvme12n4 --name=job-/dev/nvme4n4 --filename=/dev/nvme4n4 --name=job-/dev/nvme8n5 --filename=/dev/nvme8n5 --name=job-/dev/nvme12n5 --filename=/dev/nvme12n5 --name=job-/dev/nvme4n5 --filename=/dev/nvme4n5 --name=job-/dev/nvme8n6 --filename=/dev/nvme8n6 --name=job-/dev/nvme12n6 --filename=/dev/nvme12n6 --name=job-/dev/nvme4n6 --filename=/dev/nvme4n6 --name=job-/dev/nvme8n7 --filename=/dev/nvme8n7 --name=job-/dev/nvme12n7 --filename=/dev/nvme12n7 --name=job-/dev/nvme4n7 --filename=/dev/nvme4n7 --name=job-/dev/nvme8n8 --filename=/dev/nvme8n8 --name=job-/dev/nvme12n8 --filename=/dev/nvme12n8 --name=job-/dev/nvme4n8 --filename=/dev/nvme4n8 --name=job-/dev/nvme8n9 --filename=/dev/nvme8n9 --name=job-/dev/nvme12n9 --filename=/dev/nvme12n9 --name=job-/dev/nvme4n9 --filename=/dev/nvme4n9 --name=job-/dev/nvme8n10 --filename=/dev/nvme8n10 --name=job-/dev/nvme12n10 --filename=/dev/nvme12n10 --name=job-/dev/nvme4n10 --filename=/dev/nvme4n10 --name=job-/dev/nvme8n11 --filename=/dev/nvme8n11 --name=job-/dev/nvme12n11 --filename=/dev/nvme12n11 --name=job-/dev/nvme4n11 --filename=/dev/nvme4n11 --name=job-/dev/nvme8n12 --filename=/dev/nvme8n12 --name=job-/dev/nvme12n12 --filename=/dev/nvme12n12 --name=job-/dev/nvme4n12 --filename=/dev/nvme4n12 --name=job-/dev/nvme8n13 --filename=/dev/nvme8n13 --name=job-/dev/nvme12n13 --filename=/dev/nvme12n13 --name=job-/dev/nvme4n13 --filename=/dev/nvme4n13 --name=job-/dev/nvme8n14 --filename=/dev/nvme8n14 --name=job-/dev/nvme12n14 --filename=/dev/nvme12n14 --name=job-/dev/nvme4n14 --filename=/dev/nvme4n14 --name=job-/dev/nvme8n15 --filename=/dev/nvme8n15 --name=job-/dev/nvme12n15 --filename=/dev/nvme12n15 --name=job-/dev/nvme4n15 --filename=/dev/nvme4n15 --name=job-/dev/nvme8n16 --filename=/dev/nvme8n16 --name=job-/dev/nvme12n16 --filename=/dev/nvme12n16 --name=job-/dev/nvme4n16 --filename=/dev/nvme4n16 --name=job-/dev/nvme8n17 --filename=/dev/nvme8n17 --name=job-/dev/nvme12n17 --filename=/dev/nvme12n17 --name=job-/dev/nvme4n17 --filename=/dev/nvme4n17 --name=job-/dev/nvme8n18 --filename=/dev/nvme8n18 --name=job-/dev/nvme12n18 --filename=/dev/nvme12n18 --name=job-/dev/nvme4n18 --filename=/dev/nvme4n18 --name=job-/dev/nvme8n19 --filename=/dev/nvme8n19 --name=job-/dev/nvme12n19 --filename=/dev/nvme12n19 --name=job-/dev/nvme4n19 --filename=/dev/nvme4n19 --name=job-/dev/nvme8n20 --filename=/dev/nvme8n20 --name=job-/dev/nvme12n20 --filename=/dev/nvme12n20 --name=job-/dev/nvme4n20 --filename=/dev/nvme4n20 2026-02-24T16:48:56.245 INFO:tasks.workunit.client.0.trial051.stderr:+ set +e 2026-02-24T16:48:56.245 INFO:tasks.workunit.client.0.trial051.stderr:+ sudo fio /tmp/nvmeof-fio-W4pN 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n1: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n1: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n1: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n2: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n2: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n2: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n3: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n3: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n3: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.258 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n4: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n4: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n4: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n5: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n5: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n5: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n6: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n6: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n6: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n7: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n7: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n7: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n8: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n8: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n8: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n9: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n9: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n9: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.259 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n10: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.260 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n10: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.260 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n10: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.260 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n11: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.260 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n11: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.260 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n11: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.399 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n12: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n12: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n12: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n13: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n13: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n13: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n14: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n14: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n14: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.400 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n15: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n15: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n15: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n16: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n16: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n16: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n17: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n17: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n17: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n18: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n18: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n18: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n19: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.401 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n19: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n19: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n20: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme12n20: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme4n20: (g=0): rw=randrw, bs=(R) 4096B-64.0KiB, (W) 4096B-64.0KiB, (T) 4096B-64.0KiB, ioengine=sync, iodepth=1 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:fio-3.35 2026-02-24T16:48:56.402 INFO:tasks.workunit.client.0.trial051.stdout:Starting 60 processes 2026-02-24T16:48:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:48:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: { 2026-02-24T16:48:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: "prefix": "rbd perf image stats", 2026-02-24T16:48:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: "pool_spec": "mypool", 2026-02-24T16:48:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: "sort_by": "write_ops", 2026-02-24T16:48:58.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: "format": "json" 2026-02-24T16:48:58.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: }"]: dispatch 2026-02-24T16:48:58.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:57 trial153 ceph-mon[44017]: pgmap v277: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2026-02-24T16:48:58.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:48:58.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: { 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: "prefix": "rbd perf image stats", 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: "pool_spec": "mypool", 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: "sort_by": "write_ops", 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: "format": "json" 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: }"]: dispatch 2026-02-24T16:48:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:57 trial151 ceph-mon[39453]: pgmap v277: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: { 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: "prefix": "rbd perf image stats", 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: "pool_spec": "mypool", 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: "sort_by": "write_ops", 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: "format": "json" 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: }"]: dispatch 2026-02-24T16:48:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:57 trial183 ceph-mon[43553]: pgmap v277: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2026-02-24T16:48:59.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2945600797' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/4195800290' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/558828011' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:58 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2784679101' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2945600797' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/4195800290' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/558828011' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:58 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2784679101' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2945600797' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/4195800290' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/558828011' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:48:59.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:58 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2784679101' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:48:59 trial153 ceph-mon[44017]: pgmap v278: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 2.2 KiB/s wr, 16 op/s 2026-02-24T16:49:00.134 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:48:59 trial151 ceph-mon[39453]: pgmap v278: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 2.2 KiB/s wr, 16 op/s 2026-02-24T16:49:00.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:48:59 trial183 ceph-mon[43553]: pgmap v278: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 2.2 KiB/s wr, 16 op/s 2026-02-24T16:49:02.035 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:01 trial183 ceph-mon[43553]: pgmap v279: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2026-02-24T16:49:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:01 trial153 ceph-mon[44017]: pgmap v279: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2026-02-24T16:49:02.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:01 trial151 ceph-mon[39453]: pgmap v279: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2026-02-24T16:49:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: { 2026-02-24T16:49:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: "prefix": "rbd perf image stats", 2026-02-24T16:49:03.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: "pool_spec": "mypool", 2026-02-24T16:49:03.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: "sort_by": "write_ops", 2026-02-24T16:49:03.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: "format": "json" 2026-02-24T16:49:03.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:02 trial153 ceph-mon[44017]: }"]: dispatch 2026-02-24T16:49:03.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:03.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: { 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: "prefix": "rbd perf image stats", 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: "pool_spec": "mypool", 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: "sort_by": "write_ops", 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: "format": "json" 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:02 trial151 ceph-mon[39453]: }"]: dispatch 2026-02-24T16:49:03.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: { 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: "prefix": "rbd perf image stats", 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: "pool_spec": "mypool", 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: "sort_by": "write_ops", 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: "format": "json" 2026-02-24T16:49:03.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:02 trial183 ceph-mon[43553]: }"]: dispatch 2026-02-24T16:49:04.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:03 trial153 ceph-mon[44017]: pgmap v280: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2026-02-24T16:49:04.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:03 trial151 ceph-mon[39453]: pgmap v280: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2026-02-24T16:49:04.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:03 trial183 ceph-mon[43553]: pgmap v280: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 70 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2026-02-24T16:49:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:05 trial153 ceph-mon[44017]: pgmap v281: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 5.0 KiB/s wr, 17 op/s 2026-02-24T16:49:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2898981352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1796773722' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2379135710' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:05 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1570341137' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:06.117 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:06.117 INFO:tasks.workunit.client.0.trial051.stdout: 110.70 1.8M 1.9M 0.0k 18.2M 18.8M 0.0k nvme12n1 2026-02-24T16:49:06.117 INFO:tasks.workunit.client.0.trial051.stdout: 123.80 2.1M 1.8M 0.0k 20.8M 18.4M 0.0k nvme12n10 2026-02-24T16:49:06.117 INFO:tasks.workunit.client.0.trial051.stdout: 121.70 2.1M 1.9M 0.0k 20.9M 19.4M 0.0k nvme12n11 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.50 2.1M 1.8M 0.0k 21.1M 18.0M 0.0k nvme12n12 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 111.00 2.0M 1.6M 0.0k 19.5M 16.5M 0.0k nvme12n13 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 95.40 1.6M 1.4M 0.0k 16.5M 14.5M 0.0k nvme12n14 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 120.80 2.0M 1.9M 0.0k 20.0M 18.7M 0.0k nvme12n15 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 120.00 2.0M 1.9M 0.0k 19.7M 18.5M 0.0k nvme12n16 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 2.0M 1.9M 0.0k 20.3M 18.8M 0.0k nvme12n17 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 102.80 1.6M 1.6M 0.0k 16.3M 16.4M 0.0k nvme12n18 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 2.0M 1.8M 0.0k 19.8M 18.2M 0.0k nvme12n19 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 100.80 1.6M 1.5M 0.0k 16.2M 15.4M 0.0k nvme12n2 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 117.20 2.0M 1.9M 0.0k 19.9M 19.2M 0.0k nvme12n20 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 105.40 1.7M 1.8M 0.0k 17.0M 17.9M 0.0k nvme12n3 2026-02-24T16:49:06.118 INFO:tasks.workunit.client.0.trial051.stdout: 130.30 2.2M 2.0M 0.0k 21.9M 20.3M 0.0k nvme12n4 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 2.0M 1.9M 0.0k 19.6M 18.7M 0.0k nvme12n5 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 118.60 2.1M 1.8M 0.0k 20.7M 18.1M 0.0k nvme12n6 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.00 1.9M 1.6M 0.0k 18.8M 16.1M 0.0k nvme12n7 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.20 1.9M 1.6M 0.0k 19.3M 16.0M 0.0k nvme12n8 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 94.00 1.5M 1.4M 0.0k 15.1M 13.9M 0.0k nvme12n9 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 2.0M 1.9M 0.0k 20.2M 18.9M 0.0k nvme4n1 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 127.90 2.1M 1.9M 0.0k 21.0M 19.2M 0.0k nvme4n10 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 112.80 1.8M 1.8M 0.0k 17.6M 18.5M 0.0k nvme4n11 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 89.90 1.6M 1.3M 0.0k 16.0M 13.4M 0.0k nvme4n12 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 2.0M 2.0M 0.0k 20.5M 20.0M 0.0k nvme4n13 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 119.50 2.0M 1.9M 0.0k 19.7M 19.1M 0.0k nvme4n14 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.10 1.8M 1.7M 0.0k 18.3M 17.3M 0.0k nvme4n15 2026-02-24T16:49:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 109.50 1.8M 1.8M 0.0k 18.4M 18.2M 0.0k nvme4n16 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 108.20 1.8M 1.7M 0.0k 18.3M 16.8M 0.0k nvme4n17 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 112.20 1.8M 1.8M 0.0k 18.1M 18.4M 0.0k nvme4n18 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 106.20 1.8M 1.8M 0.0k 17.5M 17.9M 0.0k nvme4n19 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 106.60 1.8M 1.7M 0.0k 18.1M 16.7M 0.0k nvme4n2 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 130.70 2.2M 2.0M 0.0k 22.3M 19.6M 0.0k nvme4n20 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 81.60 1.3M 1.3M 0.0k 12.6M 13.1M 0.0k nvme4n3 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 120.50 2.0M 1.9M 0.0k 20.4M 18.9M 0.0k nvme4n4 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 1.9M 1.9M 0.0k 19.1M 18.6M 0.0k nvme4n5 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 120.90 2.0M 2.0M 0.0k 20.0M 19.7M 0.0k nvme4n6 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 127.90 2.2M 2.1M 0.0k 22.1M 21.4M 0.0k nvme4n7 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 105.40 1.6M 1.6M 0.0k 16.4M 16.2M 0.0k nvme4n8 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 123.40 2.0M 2.0M 0.0k 19.6M 20.0M 0.0k nvme4n9 2026-02-24T16:49:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 104.60 1.6M 1.6M 0.0k 16.3M 16.4M 0.0k nvme8n1 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 120.70 1.9M 1.9M 0.0k 19.0M 18.9M 0.0k nvme8n10 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 113.20 1.8M 1.8M 0.0k 18.4M 18.3M 0.0k nvme8n11 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 107.30 1.7M 1.6M 0.0k 16.8M 16.3M 0.0k nvme8n12 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 89.10 1.5M 1.5M 0.0k 14.5M 14.9M 0.0k nvme8n13 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 80.50 1.3M 1.4M 0.0k 12.9M 13.8M 0.0k nvme8n14 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 97.30 1.5M 1.5M 0.0k 15.4M 15.4M 0.0k nvme8n15 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 126.00 2.1M 2.0M 0.0k 20.8M 19.7M 0.0k nvme8n16 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 133.10 2.2M 2.1M 0.0k 22.0M 20.7M 0.0k nvme8n17 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 98.40 1.6M 1.6M 0.0k 15.8M 15.9M 0.0k nvme8n18 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 116.60 1.9M 1.8M 0.0k 18.9M 18.4M 0.0k nvme8n19 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 111.40 1.8M 1.8M 0.0k 17.8M 17.8M 0.0k nvme8n2 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 107.20 1.8M 1.7M 0.0k 18.2M 16.6M 0.0k nvme8n20 2026-02-24T16:49:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 129.00 2.1M 2.0M 0.0k 21.3M 20.4M 0.0k nvme8n3 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 116.90 2.0M 1.9M 0.0k 20.1M 18.6M 0.0k nvme8n4 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 122.40 2.0M 1.8M 0.0k 20.2M 18.3M 0.0k nvme8n5 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 100.30 1.6M 1.6M 0.0k 15.6M 15.7M 0.0k nvme8n6 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 106.20 1.7M 1.6M 0.0k 17.5M 16.3M 0.0k nvme8n7 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 100.10 1.5M 1.6M 0.0k 15.3M 15.8M 0.0k nvme8n8 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 128.00 2.1M 1.9M 0.0k 20.8M 19.1M 0.0k nvme8n9 2026-02-24T16:49:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:06.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:05 trial151 ceph-mon[39453]: pgmap v281: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 5.0 KiB/s wr, 17 op/s 2026-02-24T16:49:06.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2898981352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1796773722' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2379135710' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:05 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1570341137' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:05 trial183 ceph-mon[43553]: pgmap v281: 33 pgs: 33 active+clean; 618 KiB data, 110 MiB used, 2.7 TiB / 2.7 TiB avail; 47 KiB/s rd, 5.0 KiB/s wr, 17 op/s 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2898981352' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1796773722' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2379135710' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:06.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:05 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1570341137' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: { 2026-02-24T16:49:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: "prefix": "rbd perf image stats", 2026-02-24T16:49:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: "pool_spec": "mypool", 2026-02-24T16:49:08.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: "sort_by": "write_ops", 2026-02-24T16:49:08.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: "format": "json" 2026-02-24T16:49:08.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: }"]: dispatch 2026-02-24T16:49:08.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:07 trial153 ceph-mon[44017]: pgmap v282: 33 pgs: 33 active+clean; 26 GiB data, 2.2 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s rd, 65 MiB/s wr, 9.94k op/s 2026-02-24T16:49:08.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:08.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: { 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: "prefix": "rbd perf image stats", 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: "pool_spec": "mypool", 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: "sort_by": "write_ops", 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: "format": "json" 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: }"]: dispatch 2026-02-24T16:49:08.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:07 trial151 ceph-mon[39453]: pgmap v282: 33 pgs: 33 active+clean; 26 GiB data, 2.2 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s rd, 65 MiB/s wr, 9.94k op/s 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: { 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: "prefix": "rbd perf image stats", 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: "pool_spec": "mypool", 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: "sort_by": "write_ops", 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: "format": "json" 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: }"]: dispatch 2026-02-24T16:49:08.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:07 trial183 ceph-mon[43553]: pgmap v282: 33 pgs: 33 active+clean; 26 GiB data, 2.2 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s rd, 65 MiB/s wr, 9.94k op/s 2026-02-24T16:49:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:09 trial153 ceph-mon[44017]: pgmap v283: 33 pgs: 33 active+clean; 35 GiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s rd, 94 MiB/s wr, 13.54k op/s 2026-02-24T16:49:10.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:09 trial151 ceph-mon[39453]: pgmap v283: 33 pgs: 33 active+clean; 35 GiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s rd, 94 MiB/s wr, 13.54k op/s 2026-02-24T16:49:10.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:09 trial183 ceph-mon[43553]: pgmap v283: 33 pgs: 33 active+clean; 35 GiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s rd, 94 MiB/s wr, 13.54k op/s 2026-02-24T16:49:11.143 INFO:tasks.workunit.client.0.trial051.stdout:NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT 2026-02-24T16:49:11.143 INFO:tasks.workunit.client.0.trial051.stdout:myimage7 217/s 188/s 7.1 MiB/s 4.1 MiB/s 2.44 ms 723.27 us 2026-02-24T16:49:11.143 INFO:tasks.workunit.client.0.trial051.stdout:myimage10 204/s 173/s 6.4 MiB/s 3.6 MiB/s 2.35 ms 820.58 us 2026-02-24T16:49:11.143 INFO:tasks.workunit.client.0.trial051.stdout:myimage55 202/s 175/s 6.4 MiB/s 3.7 MiB/s 2.49 ms 794.24 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage44 201/s 173/s 6.5 MiB/s 3.8 MiB/s 2.45 ms 760.25 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage36 200/s 170/s 6.6 MiB/s 3.4 MiB/s 2.39 ms 726.79 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage20 197/s 186/s 6.3 MiB/s 3.8 MiB/s 2.47 ms 770.80 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage23 194/s 165/s 6.1 MiB/s 3.3 MiB/s 2.44 ms 861.39 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage19 192/s 158/s 6.2 MiB/s 3.4 MiB/s 2.44 ms 789.57 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage51 192/s 163/s 6.2 MiB/s 3.4 MiB/s 2.39 ms 828.83 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage9 191/s 173/s 6.2 MiB/s 3.8 MiB/s 2.48 ms 780.39 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage58 191/s 155/s 6.3 MiB/s 3.2 MiB/s 2.69 ms 837.82 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage59 191/s 158/s 6.4 MiB/s 3.3 MiB/s 2.58 ms 881.66 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage60 190/s 165/s 6.1 MiB/s 3.4 MiB/s 2.60 ms 797.49 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage24 190/s 165/s 6.2 MiB/s 3.5 MiB/s 2.59 ms 974.63 us 2026-02-24T16:49:11.144 INFO:tasks.workunit.client.0.trial051.stdout:myimage5 189/s 156/s 6.0 MiB/s 3.3 MiB/s 2.59 ms 802.36 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage17 187/s 174/s 5.9 MiB/s 3.5 MiB/s 2.51 ms 702.86 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage22 187/s 153/s 5.9 MiB/s 3.1 MiB/s 2.57 ms 789.00 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage6 186/s 160/s 6.1 MiB/s 3.5 MiB/s 2.49 ms 903.43 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage47 185/s 153/s 6.0 MiB/s 3.1 MiB/s 2.54 ms 813.69 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage48 185/s 152/s 6.1 MiB/s 3.3 MiB/s 2.62 ms 775.37 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage21 184/s 151/s 5.7 MiB/s 2.8 MiB/s 2.44 ms 745.37 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage37 184/s 152/s 5.9 MiB/s 3.3 MiB/s 2.61 ms 905.55 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage14 184/s 158/s 5.9 MiB/s 3.3 MiB/s 2.58 ms 802.35 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage29 183/s 178/s 5.9 MiB/s 3.9 MiB/s 2.57 ms 743.77 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage57 182/s 144/s 5.9 MiB/s 3.0 MiB/s 2.57 ms 963.62 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage39 181/s 156/s 5.7 MiB/s 2.9 MiB/s 2.66 ms 868.76 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage4 181/s 162/s 5.9 MiB/s 3.5 MiB/s 2.67 ms 822.36 us 2026-02-24T16:49:11.145 INFO:tasks.workunit.client.0.trial051.stdout:myimage32 180/s 157/s 5.7 MiB/s 3.1 MiB/s 2.55 ms 784.72 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage30 180/s 153/s 5.7 MiB/s 3.1 MiB/s 2.60 ms 852.25 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage50 179/s 158/s 5.8 MiB/s 3.4 MiB/s 2.63 ms 845.52 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage15 179/s 158/s 5.8 MiB/s 3.3 MiB/s 2.55 ms 814.26 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage31 178/s 147/s 5.8 MiB/s 3.1 MiB/s 2.68 ms 919.85 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage1 178/s 145/s 5.8 MiB/s 3.0 MiB/s 2.54 ms 935.31 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage27 178/s 154/s 5.7 MiB/s 3.0 MiB/s 2.55 ms 878.19 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage52 178/s 157/s 5.7 MiB/s 3.3 MiB/s 2.51 ms 810.70 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage8 177/s 148/s 5.7 MiB/s 3.0 MiB/s 2.50 ms 883.69 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage43 177/s 144/s 5.7 MiB/s 2.8 MiB/s 2.52 ms 642.47 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage46 176/s 161/s 5.8 MiB/s 3.4 MiB/s 2.59 ms 900.67 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage11 176/s 149/s 5.5 MiB/s 3.1 MiB/s 2.55 ms 856.87 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage38 175/s 158/s 5.6 MiB/s 3.1 MiB/s 2.58 ms 859.89 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage2 175/s 153/s 5.6 MiB/s 3.3 MiB/s 2.63 ms 910.85 us 2026-02-24T16:49:11.146 INFO:tasks.workunit.client.0.trial051.stdout:myimage16 175/s 146/s 5.8 MiB/s 3.2 MiB/s 2.64 ms 885.09 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage25 174/s 142/s 5.6 MiB/s 2.9 MiB/s 2.45 ms 917.85 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage41 174/s 138/s 5.9 MiB/s 2.9 MiB/s 2.81 ms 1.01 ms 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage56 174/s 158/s 5.5 MiB/s 3.1 MiB/s 2.62 ms 814.18 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage45 174/s 151/s 5.7 MiB/s 3.3 MiB/s 2.64 ms 862.11 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage26 171/s 153/s 5.3 MiB/s 3.3 MiB/s 2.50 ms 829.09 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage13 171/s 147/s 5.6 MiB/s 3.0 MiB/s 2.61 ms 883.06 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage42 171/s 145/s 5.5 MiB/s 2.9 MiB/s 2.64 ms 903.93 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage28 168/s 147/s 5.4 MiB/s 2.9 MiB/s 2.81 ms 930.48 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage18 166/s 140/s 5.3 MiB/s 2.9 MiB/s 2.63 ms 1.01 ms 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage35 165/s 134/s 5.4 MiB/s 2.7 MiB/s 2.74 ms 1.06 ms 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage53 162/s 133/s 5.0 MiB/s 2.5 MiB/s 2.70 ms 1.03 ms 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage40 161/s 154/s 5.3 MiB/s 3.1 MiB/s 2.53 ms 983.02 us 2026-02-24T16:49:11.147 INFO:tasks.workunit.client.0.trial051.stdout:myimage33 159/s 120/s 5.3 MiB/s 2.4 MiB/s 2.61 ms 965.78 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout:myimage3 158/s 126/s 5.1 MiB/s 2.8 MiB/s 2.51 ms 813.58 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout:myimage49 155/s 123/s 4.9 MiB/s 2.4 MiB/s 2.48 ms 841.12 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout:myimage54 153/s 115/s 5.0 MiB/s 2.4 MiB/s 2.59 ms 778.87 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout:myimage12 151/s 125/s 5.0 MiB/s 2.6 MiB/s 2.67 ms 805.24 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout:myimage34 149/s 132/s 4.6 MiB/s 2.5 MiB/s 2.56 ms 860.71 us 2026-02-24T16:49:11.148 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:12.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:11 trial153 ceph-mon[44017]: pgmap v284: 33 pgs: 33 active+clean; 41 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 141 MiB/s wr, 17.08k op/s 2026-02-24T16:49:12.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:11 trial151 ceph-mon[39453]: pgmap v284: 33 pgs: 33 active+clean; 41 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 141 MiB/s wr, 17.08k op/s 2026-02-24T16:49:12.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:11 trial183 ceph-mon[43553]: pgmap v284: 33 pgs: 33 active+clean; 41 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 141 MiB/s wr, 17.08k op/s 2026-02-24T16:49:13.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:13.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: { 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: "prefix": "rbd perf image stats", 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: "pool_spec": "mypool", 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: "sort_by": "write_ops", 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: "format": "json" 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: }"]: dispatch 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1770891070' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2619136738' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3024524488' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:12 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/750297640' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:13.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: { 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: "prefix": "rbd perf image stats", 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: "pool_spec": "mypool", 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: "sort_by": "write_ops", 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: "format": "json" 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: }"]: dispatch 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1770891070' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2619136738' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3024524488' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:12 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/750297640' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: from='client.37446 -' entity='client.admin' cmd=[ 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: { 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: "prefix": "rbd perf image stats", 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: "pool_spec": "mypool", 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: "sort_by": "write_ops", 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: "format": "json" 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: }"]: dispatch 2026-02-24T16:49:13.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1770891070' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2619136738' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3024524488' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:13.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:12 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/750297640' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:13 trial153 ceph-mon[44017]: pgmap v285: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.71k op/s 2026-02-24T16:49:14.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:13 trial151 ceph-mon[39453]: pgmap v285: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.71k op/s 2026-02-24T16:49:14.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:13 trial183 ceph-mon[43553]: pgmap v285: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.71k op/s 2026-02-24T16:49:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:15 trial153 ceph-mon[44017]: pgmap v286: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.70k op/s 2026-02-24T16:49:16.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:16.119 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 321.00 5.1M 5.3M 0.0k 50.6M 53.3M 0.0k nvme12n1 2026-02-24T16:49:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 346.50 5.5M 5.5M 0.0k 55.4M 55.5M 0.0k nvme12n10 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 360.90 5.6M 5.9M 0.0k 56.3M 58.9M 0.0k nvme12n11 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 348.20 5.4M 5.5M 0.0k 54.2M 54.5M 0.0k nvme12n12 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 310.60 4.9M 4.9M 0.0k 49.3M 49.0M 0.0k nvme12n13 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 333.30 5.5M 5.3M 0.0k 55.0M 53.4M 0.0k nvme12n14 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 370.50 5.8M 5.8M 0.0k 57.9M 58.5M 0.0k nvme12n15 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 348.10 5.4M 5.3M 0.0k 53.8M 52.8M 0.0k nvme12n16 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 339.60 5.4M 5.3M 0.0k 53.7M 53.1M 0.0k nvme12n17 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 339.30 5.4M 5.5M 0.0k 54.1M 54.8M 0.0k nvme12n18 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 341.20 5.6M 5.5M 0.0k 55.9M 54.9M 0.0k nvme12n19 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 333.10 5.3M 5.2M 0.0k 53.4M 52.2M 0.0k nvme12n2 2026-02-24T16:49:16.120 INFO:tasks.workunit.client.0.trial051.stdout: 350.40 5.6M 5.5M 0.0k 55.5M 54.5M 0.0k nvme12n20 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 354.10 5.5M 5.6M 0.0k 55.1M 55.5M 0.0k nvme12n3 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 378.90 6.0M 5.8M 0.0k 60.2M 57.9M 0.0k nvme12n4 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 325.40 5.0M 5.2M 0.0k 50.2M 52.5M 0.0k nvme12n5 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 333.20 5.3M 5.2M 0.0k 52.5M 51.6M 0.0k nvme12n6 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 361.50 5.7M 6.0M 0.0k 56.6M 59.8M 0.0k nvme12n7 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 356.70 5.5M 5.8M 0.0k 55.0M 57.8M 0.0k nvme12n8 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 319.20 4.9M 5.2M 0.0k 49.4M 52.2M 0.0k nvme12n9 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 350.30 5.3M 5.7M 0.0k 53.2M 56.9M 0.0k nvme4n1 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 382.60 5.8M 6.0M 0.0k 58.5M 60.3M 0.0k nvme4n10 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 343.30 5.5M 5.3M 0.0k 54.6M 52.9M 0.0k nvme4n11 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 315.90 5.0M 5.1M 0.0k 50.2M 50.5M 0.0k nvme4n12 2026-02-24T16:49:16.121 INFO:tasks.workunit.client.0.trial051.stdout: 314.20 5.1M 5.1M 0.0k 51.3M 50.6M 0.0k nvme4n13 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 343.70 5.4M 5.4M 0.0k 53.8M 53.9M 0.0k nvme4n14 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 363.30 5.8M 5.7M 0.0k 58.0M 56.5M 0.0k nvme4n15 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 331.80 5.2M 5.3M 0.0k 52.3M 53.2M 0.0k nvme4n16 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 382.50 6.2M 5.8M 0.0k 61.8M 58.1M 0.0k nvme4n17 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 314.90 5.0M 5.0M 0.0k 50.3M 49.7M 0.0k nvme4n18 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 362.70 5.6M 5.8M 0.0k 56.3M 57.8M 0.0k nvme4n19 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 343.80 5.6M 5.2M 0.0k 56.1M 52.0M 0.0k nvme4n2 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 373.10 6.0M 5.7M 0.0k 60.3M 57.0M 0.0k nvme4n20 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 336.30 5.4M 5.3M 0.0k 53.9M 53.5M 0.0k nvme4n3 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 349.40 5.5M 5.6M 0.0k 55.0M 55.7M 0.0k nvme4n4 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 355.50 5.7M 5.6M 0.0k 56.7M 56.3M 0.0k nvme4n5 2026-02-24T16:49:16.122 INFO:tasks.workunit.client.0.trial051.stdout: 346.50 5.4M 5.5M 0.0k 54.2M 55.0M 0.0k nvme4n6 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 393.40 6.1M 6.1M 0.0k 61.5M 61.0M 0.0k nvme4n7 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 349.80 5.6M 5.4M 0.0k 55.6M 53.7M 0.0k nvme4n8 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 359.10 5.8M 5.5M 0.0k 58.5M 55.5M 0.0k nvme4n9 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 361.50 5.5M 5.7M 0.0k 54.8M 57.4M 0.0k nvme8n1 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 342.30 5.4M 5.2M 0.0k 53.8M 52.1M 0.0k nvme8n10 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 328.30 5.2M 5.2M 0.0k 51.6M 51.7M 0.0k nvme8n11 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 358.10 5.6M 5.9M 0.0k 55.6M 58.7M 0.0k nvme8n12 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 326.60 5.2M 5.3M 0.0k 52.5M 53.2M 0.0k nvme8n13 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 334.50 5.3M 5.1M 0.0k 53.4M 51.0M 0.0k nvme8n14 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 314.30 5.0M 5.0M 0.0k 49.9M 50.4M 0.0k nvme8n15 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 387.80 6.0M 5.9M 0.0k 60.5M 59.3M 0.0k nvme8n16 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 337.30 5.2M 5.3M 0.0k 51.6M 53.4M 0.0k nvme8n17 2026-02-24T16:49:16.123 INFO:tasks.workunit.client.0.trial051.stdout: 361.30 5.8M 5.6M 0.0k 57.8M 55.5M 0.0k nvme8n18 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 348.50 5.5M 5.4M 0.0k 55.1M 53.9M 0.0k nvme8n19 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 359.70 5.6M 5.7M 0.0k 55.9M 56.7M 0.0k nvme8n2 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 320.30 5.3M 4.9M 0.0k 52.7M 49.4M 0.0k nvme8n20 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 356.90 5.6M 5.6M 0.0k 56.0M 55.9M 0.0k nvme8n3 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 364.80 5.9M 5.5M 0.0k 58.7M 55.0M 0.0k nvme8n4 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 345.10 5.5M 5.4M 0.0k 54.6M 54.3M 0.0k nvme8n5 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 360.20 5.7M 5.5M 0.0k 57.0M 55.4M 0.0k nvme8n6 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 347.90 5.5M 5.2M 0.0k 55.4M 52.2M 0.0k nvme8n7 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 327.20 5.2M 5.0M 0.0k 52.3M 50.2M 0.0k nvme8n8 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 351.60 5.7M 5.6M 0.0k 57.4M 55.7M 0.0k nvme8n9 2026-02-24T16:49:16.124 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:16.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:15 trial151 ceph-mon[39453]: pgmap v286: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.70k op/s 2026-02-24T16:49:16.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:15 trial183 ceph-mon[43553]: pgmap v286: 33 pgs: 33 active+clean; 51 GiB data, 9.6 GiB used, 2.7 TiB / 2.7 TiB avail; 143 MiB/s rd, 253 MiB/s wr, 24.70k op/s 2026-02-24T16:49:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:17 trial153 ceph-mon[44017]: pgmap v287: 33 pgs: 33 active+clean; 53 GiB data, 13 GiB used, 2.7 TiB / 2.7 TiB avail; 212 MiB/s rd, 343 MiB/s wr, 30.54k op/s 2026-02-24T16:49:18.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:17 trial151 ceph-mon[39453]: pgmap v287: 33 pgs: 33 active+clean; 53 GiB data, 13 GiB used, 2.7 TiB / 2.7 TiB avail; 212 MiB/s rd, 343 MiB/s wr, 30.54k op/s 2026-02-24T16:49:18.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:17 trial183 ceph-mon[43553]: pgmap v287: 33 pgs: 33 active+clean; 53 GiB data, 13 GiB used, 2.7 TiB / 2.7 TiB avail; 212 MiB/s rd, 343 MiB/s wr, 30.54k op/s 2026-02-24T16:49:20.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:19 trial153 ceph-mon[44017]: pgmap v288: 33 pgs: 33 active+clean; 54 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 207 MiB/s rd, 300 MiB/s wr, 22.10k op/s 2026-02-24T16:49:20.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:19 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/713885912' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:19 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1319136051' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:19 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1039265463' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:19 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3635384556' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:19 trial151 ceph-mon[39453]: pgmap v288: 33 pgs: 33 active+clean; 54 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 207 MiB/s rd, 300 MiB/s wr, 22.10k op/s 2026-02-24T16:49:20.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/713885912' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1319136051' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1039265463' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:19 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3635384556' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:19 trial183 ceph-mon[43553]: pgmap v288: 33 pgs: 33 active+clean; 54 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 207 MiB/s rd, 300 MiB/s wr, 22.10k op/s 2026-02-24T16:49:20.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:19 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/713885912' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:19 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1319136051' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:19 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1039265463' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:20.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:19 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3635384556' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:21 trial153 ceph-mon[44017]: pgmap v289: 33 pgs: 33 active+clean; 54 GiB data, 15 GiB used, 2.7 TiB / 2.7 TiB avail; 213 MiB/s rd, 296 MiB/s wr, 20.11k op/s 2026-02-24T16:49:22.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:21 trial151 ceph-mon[39453]: pgmap v289: 33 pgs: 33 active+clean; 54 GiB data, 15 GiB used, 2.7 TiB / 2.7 TiB avail; 213 MiB/s rd, 296 MiB/s wr, 20.11k op/s 2026-02-24T16:49:22.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:21 trial183 ceph-mon[43553]: pgmap v289: 33 pgs: 33 active+clean; 54 GiB data, 15 GiB used, 2.7 TiB / 2.7 TiB avail; 213 MiB/s rd, 296 MiB/s wr, 20.11k op/s 2026-02-24T16:49:23.483 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:23 trial151 ceph-mon[39453]: pgmap v290: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 233 MiB/s rd, 310 MiB/s wr, 20.61k op/s 2026-02-24T16:49:23.484 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:23 trial183 ceph-mon[43553]: pgmap v290: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 233 MiB/s rd, 310 MiB/s wr, 20.61k op/s 2026-02-24T16:49:23.564 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:23 trial153 ceph-mon[44017]: pgmap v290: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 233 MiB/s rd, 310 MiB/s wr, 20.61k op/s 2026-02-24T16:49:25.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:25 trial151 ceph-mon[39453]: pgmap v291: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 157 MiB/s rd, 198 MiB/s wr, 12.98k op/s 2026-02-24T16:49:25.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:25 trial183 ceph-mon[43553]: pgmap v291: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 157 MiB/s rd, 198 MiB/s wr, 12.98k op/s 2026-02-24T16:49:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:25 trial153 ceph-mon[44017]: pgmap v291: 33 pgs: 33 active+clean; 55 GiB data, 18 GiB used, 2.7 TiB / 2.7 TiB avail; 157 MiB/s rd, 198 MiB/s wr, 12.98k op/s 2026-02-24T16:49:26.105 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 180.40 2.7M 2.8M 0.0k 26.8M 28.5M 0.0k nvme12n1 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 207.50 3.4M 3.1M 0.0k 34.0M 30.5M 0.0k nvme12n10 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 223.60 3.4M 3.2M 0.0k 34.3M 32.2M 0.0k nvme12n11 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 211.70 3.2M 3.0M 0.0k 31.7M 30.4M 0.0k nvme12n12 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 188.20 2.7M 2.9M 0.0k 27.0M 29.5M 0.0k nvme12n13 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 211.30 3.3M 3.3M 0.0k 33.3M 33.1M 0.0k nvme12n14 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 218.90 3.1M 3.3M 0.0k 31.4M 32.9M 0.0k nvme12n15 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 200.30 3.1M 2.9M 0.0k 30.6M 29.4M 0.0k nvme12n16 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 208.00 3.1M 3.1M 0.0k 31.1M 30.6M 0.0k nvme12n17 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 207.70 3.0M 3.1M 0.0k 30.5M 30.9M 0.0k nvme12n18 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 199.80 3.1M 2.9M 0.0k 30.7M 29.3M 0.0k nvme12n19 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 205.80 3.1M 3.1M 0.0k 31.1M 31.4M 0.0k nvme12n2 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 197.40 2.9M 3.0M 0.0k 29.3M 30.4M 0.0k nvme12n20 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 219.00 3.3M 3.4M 0.0k 33.3M 34.4M 0.0k nvme12n3 2026-02-24T16:49:26.114 INFO:tasks.workunit.client.0.trial051.stdout: 220.80 3.1M 3.4M 0.0k 31.4M 34.1M 0.0k nvme12n4 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 192.30 3.0M 2.9M 0.0k 30.2M 29.2M 0.0k nvme12n5 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 199.70 3.0M 3.0M 0.0k 30.4M 30.1M 0.0k nvme12n6 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 205.70 3.1M 3.2M 0.0k 30.5M 31.7M 0.0k nvme12n7 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 219.40 3.3M 3.3M 0.0k 32.7M 32.6M 0.0k nvme12n8 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 212.60 3.3M 3.0M 0.0k 32.9M 30.5M 0.0k nvme12n9 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 213.30 3.1M 3.1M 0.0k 30.6M 31.5M 0.0k nvme4n1 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 224.90 3.4M 3.3M 0.0k 33.7M 32.9M 0.0k nvme4n10 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 220.70 3.2M 3.3M 0.0k 32.0M 32.6M 0.0k nvme4n11 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 188.30 2.9M 3.0M 0.0k 28.8M 30.0M 0.0k nvme4n12 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 187.20 2.8M 2.9M 0.0k 28.0M 29.4M 0.0k nvme4n13 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 194.00 2.9M 2.9M 0.0k 28.8M 29.3M 0.0k nvme4n14 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 210.70 3.1M 3.2M 0.0k 30.8M 32.3M 0.0k nvme4n15 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 206.60 3.2M 3.0M 0.0k 32.2M 30.2M 0.0k nvme4n16 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 208.70 3.0M 3.0M 0.0k 30.2M 29.6M 0.0k nvme4n17 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 200.50 3.1M 3.0M 0.0k 31.3M 30.1M 0.0k nvme4n18 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 214.80 3.3M 3.1M 0.0k 32.8M 31.0M 0.0k nvme4n19 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 212.00 3.1M 3.2M 0.0k 31.1M 31.9M 0.0k nvme4n2 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 207.70 3.1M 3.2M 0.0k 30.7M 32.4M 0.0k nvme4n20 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 199.10 3.0M 3.1M 0.0k 30.1M 31.2M 0.0k nvme4n3 2026-02-24T16:49:26.115 INFO:tasks.workunit.client.0.trial051.stdout: 200.30 2.9M 3.0M 0.0k 29.2M 30.3M 0.0k nvme4n4 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 209.00 3.1M 3.1M 0.0k 31.5M 31.5M 0.0k nvme4n5 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 199.00 3.0M 2.9M 0.0k 29.9M 29.1M 0.0k nvme4n6 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 228.80 3.2M 3.4M 0.0k 32.5M 33.8M 0.0k nvme4n7 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 198.40 3.0M 3.1M 0.0k 29.7M 30.6M 0.0k nvme4n8 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 205.60 3.0M 3.2M 0.0k 30.2M 32.0M 0.0k nvme4n9 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 208.10 3.1M 3.1M 0.0k 31.1M 31.1M 0.0k nvme8n1 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 218.60 3.3M 3.3M 0.0k 33.2M 32.8M 0.0k nvme8n10 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 208.10 3.1M 3.0M 0.0k 30.7M 29.6M 0.0k nvme8n11 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 220.70 3.4M 3.4M 0.0k 33.8M 34.1M 0.0k nvme8n12 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 212.90 3.1M 3.3M 0.0k 31.3M 33.3M 0.0k nvme8n13 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 208.60 3.2M 3.2M 0.0k 32.3M 32.3M 0.0k nvme8n14 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 187.10 3.0M 2.9M 0.0k 29.7M 29.1M 0.0k nvme8n15 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 209.00 3.1M 3.2M 0.0k 30.6M 31.6M 0.0k nvme8n16 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 197.70 2.9M 3.0M 0.0k 29.4M 30.1M 0.0k nvme8n17 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 202.40 2.9M 3.0M 0.0k 29.4M 30.0M 0.0k nvme8n18 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 198.60 3.1M 2.9M 0.0k 30.6M 28.9M 0.0k nvme8n19 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 212.00 3.2M 3.1M 0.0k 31.7M 31.0M 0.0k nvme8n2 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 195.60 3.1M 2.9M 0.0k 30.5M 28.7M 0.0k nvme8n20 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 210.50 3.2M 3.1M 0.0k 31.7M 31.1M 0.0k nvme8n3 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 206.50 3.0M 3.2M 0.0k 29.6M 32.3M 0.0k nvme8n4 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 217.30 3.3M 3.2M 0.0k 33.2M 31.7M 0.0k nvme8n5 2026-02-24T16:49:26.116 INFO:tasks.workunit.client.0.trial051.stdout: 208.50 2.9M 3.0M 0.0k 29.1M 30.3M 0.0k nvme8n6 2026-02-24T16:49:26.117 INFO:tasks.workunit.client.0.trial051.stdout: 205.60 3.1M 3.1M 0.0k 30.8M 30.7M 0.0k nvme8n7 2026-02-24T16:49:26.117 INFO:tasks.workunit.client.0.trial051.stdout: 201.70 3.1M 3.1M 0.0k 30.6M 30.5M 0.0k nvme8n8 2026-02-24T16:49:26.117 INFO:tasks.workunit.client.0.trial051.stdout: 211.10 3.4M 3.2M 0.0k 33.7M 32.2M 0.0k nvme8n9 2026-02-24T16:49:26.117 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:26.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2430603770' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3571533532' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3755551704' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:26 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4011755058' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2430603770' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3571533532' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3755551704' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:26.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:26 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4011755058' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:27.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2430603770' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:27.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3571533532' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:27.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3755551704' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:27.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:26 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4011755058' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:27.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:27 trial151 ceph-mon[39453]: pgmap v292: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 202 MiB/s rd, 253 MiB/s wr, 16.73k op/s 2026-02-24T16:49:27.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:27 trial183 ceph-mon[43553]: pgmap v292: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 202 MiB/s rd, 253 MiB/s wr, 16.73k op/s 2026-02-24T16:49:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:27 trial153 ceph-mon[44017]: pgmap v292: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 202 MiB/s rd, 253 MiB/s wr, 16.73k op/s 2026-02-24T16:49:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:29 trial151 ceph-mon[39453]: pgmap v293: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 145 MiB/s rd, 178 MiB/s wr, 11.91k op/s 2026-02-24T16:49:29.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:29 trial183 ceph-mon[43553]: pgmap v293: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 145 MiB/s rd, 178 MiB/s wr, 11.91k op/s 2026-02-24T16:49:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:29 trial153 ceph-mon[44017]: pgmap v293: 33 pgs: 33 active+clean; 56 GiB data, 20 GiB used, 2.7 TiB / 2.7 TiB avail; 145 MiB/s rd, 178 MiB/s wr, 11.91k op/s 2026-02-24T16:49:31.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:31 trial151 ceph-mon[39453]: pgmap v294: 33 pgs: 33 active+clean; 56 GiB data, 21 GiB used, 2.7 TiB / 2.7 TiB avail; 142 MiB/s rd, 173 MiB/s wr, 11.62k op/s 2026-02-24T16:49:31.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:31 trial183 ceph-mon[43553]: pgmap v294: 33 pgs: 33 active+clean; 56 GiB data, 21 GiB used, 2.7 TiB / 2.7 TiB avail; 142 MiB/s rd, 173 MiB/s wr, 11.62k op/s 2026-02-24T16:49:32.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:31 trial153 ceph-mon[44017]: pgmap v294: 33 pgs: 33 active+clean; 56 GiB data, 21 GiB used, 2.7 TiB / 2.7 TiB avail; 142 MiB/s rd, 173 MiB/s wr, 11.62k op/s 2026-02-24T16:49:33.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:33 trial151 ceph-mon[39453]: pgmap v295: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 159 MiB/s rd, 192 MiB/s wr, 13.01k op/s 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:33 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1032512423' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:33 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/96168675' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:33 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3636555212' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:33 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2419579690' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:33 trial183 ceph-mon[43553]: pgmap v295: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 159 MiB/s rd, 192 MiB/s wr, 13.01k op/s 2026-02-24T16:49:33.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:33 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1032512423' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:33 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/96168675' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:33 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3636555212' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:33.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:33 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2419579690' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:33 trial153 ceph-mon[44017]: pgmap v295: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 159 MiB/s rd, 192 MiB/s wr, 13.01k op/s 2026-02-24T16:49:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:33 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1032512423' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:33 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/96168675' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:34.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:33 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3636555212' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:34.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:33 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2419579690' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:35.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:35 trial151 ceph-mon[39453]: pgmap v296: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 131 MiB/s wr, 8.97k op/s 2026-02-24T16:49:35.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:35 trial183 ceph-mon[43553]: pgmap v296: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 131 MiB/s wr, 8.97k op/s 2026-02-24T16:49:36.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:35 trial153 ceph-mon[44017]: pgmap v296: 33 pgs: 33 active+clean; 56 GiB data, 23 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 131 MiB/s wr, 8.97k op/s 2026-02-24T16:49:36.105 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 152.20 2.3M 2.2M 0.0k 23.4M 21.6M 0.0k nvme12n1 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 164.80 2.5M 2.4M 0.0k 24.7M 23.6M 0.0k nvme12n10 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 156.70 2.2M 2.3M 0.0k 22.3M 23.0M 0.0k nvme12n11 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 164.00 2.5M 2.4M 0.0k 24.7M 23.6M 0.0k nvme12n12 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 147.50 2.2M 2.2M 0.0k 22.1M 21.8M 0.0k nvme12n13 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 163.60 2.3M 2.4M 0.0k 22.8M 24.5M 0.0k nvme12n14 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 163.10 2.5M 2.4M 0.0k 24.6M 23.7M 0.0k nvme12n15 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 155.60 2.3M 2.3M 0.0k 23.3M 22.6M 0.0k nvme12n16 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 154.20 2.3M 2.1M 0.0k 23.3M 21.4M 0.0k nvme12n17 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 157.00 2.2M 2.2M 0.0k 22.4M 22.0M 0.0k nvme12n18 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 156.10 2.3M 2.4M 0.0k 22.8M 23.5M 0.0k nvme12n19 2026-02-24T16:49:36.111 INFO:tasks.workunit.client.0.trial051.stdout: 154.80 2.3M 2.4M 0.0k 22.7M 23.8M 0.0k nvme12n2 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 155.50 2.4M 2.2M 0.0k 23.5M 21.7M 0.0k nvme12n20 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 165.30 2.4M 2.5M 0.0k 24.3M 24.6M 0.0k nvme12n3 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 155.80 2.3M 2.3M 0.0k 23.0M 23.1M 0.0k nvme12n4 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 144.00 2.1M 2.1M 0.0k 20.9M 21.5M 0.0k nvme12n5 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 158.00 2.3M 2.2M 0.0k 23.1M 22.3M 0.0k nvme12n6 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 162.10 2.2M 2.3M 0.0k 21.6M 23.2M 0.0k nvme12n7 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 166.80 2.3M 2.4M 0.0k 23.1M 23.7M 0.0k nvme12n8 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 156.70 2.3M 2.3M 0.0k 22.8M 22.7M 0.0k nvme12n9 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 165.70 2.2M 2.5M 0.0k 22.3M 24.9M 0.0k nvme4n1 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 168.00 2.5M 2.4M 0.0k 24.7M 24.2M 0.0k nvme4n10 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 165.70 2.4M 2.4M 0.0k 24.0M 23.9M 0.0k nvme4n11 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 145.40 2.2M 2.2M 0.0k 21.8M 21.6M 0.0k nvme4n12 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 154.00 2.3M 2.2M 0.0k 22.9M 22.3M 0.0k nvme4n13 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 150.70 2.2M 2.2M 0.0k 21.6M 21.7M 0.0k nvme4n14 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 168.60 2.4M 2.5M 0.0k 24.5M 25.1M 0.0k nvme4n15 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 144.60 2.1M 2.2M 0.0k 20.8M 21.5M 0.0k nvme4n16 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 157.90 2.2M 2.2M 0.0k 22.5M 21.8M 0.0k nvme4n17 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 159.60 2.3M 2.3M 0.0k 23.4M 23.1M 0.0k nvme4n18 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 162.70 2.4M 2.4M 0.0k 24.2M 23.6M 0.0k nvme4n19 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 158.10 2.3M 2.3M 0.0k 23.0M 22.9M 0.0k nvme4n2 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 158.50 2.3M 2.4M 0.0k 22.7M 23.6M 0.0k nvme4n20 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 149.40 2.1M 2.2M 0.0k 21.1M 21.9M 0.0k nvme4n3 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 158.00 2.2M 2.5M 0.0k 21.9M 24.6M 0.0k nvme4n4 2026-02-24T16:49:36.112 INFO:tasks.workunit.client.0.trial051.stdout: 157.80 2.2M 2.2M 0.0k 22.0M 22.3M 0.0k nvme4n5 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 156.70 2.2M 2.4M 0.0k 22.1M 23.7M 0.0k nvme4n6 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 168.80 2.4M 2.4M 0.0k 24.1M 23.8M 0.0k nvme4n7 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 159.20 2.3M 2.2M 0.0k 23.3M 22.0M 0.0k nvme4n8 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 161.00 2.4M 2.4M 0.0k 23.6M 23.8M 0.0k nvme4n9 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 155.60 2.3M 2.2M 0.0k 22.6M 21.6M 0.0k nvme8n1 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 152.20 2.2M 2.2M 0.0k 21.8M 21.9M 0.0k nvme8n10 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 154.70 2.2M 2.2M 0.0k 21.9M 21.7M 0.0k nvme8n11 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 161.10 2.3M 2.4M 0.0k 22.6M 24.2M 0.0k nvme8n12 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 159.30 2.2M 2.4M 0.0k 22.3M 24.0M 0.0k nvme8n13 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 161.60 2.4M 2.2M 0.0k 23.9M 22.3M 0.0k nvme8n14 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 147.90 2.2M 2.2M 0.0k 21.6M 21.9M 0.0k nvme8n15 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 154.10 2.1M 2.3M 0.0k 21.2M 22.6M 0.0k nvme8n16 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 148.40 2.1M 2.1M 0.0k 21.2M 21.2M 0.0k nvme8n17 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 154.30 2.2M 2.1M 0.0k 22.3M 21.0M 0.0k nvme8n18 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 153.30 2.2M 2.1M 0.0k 21.5M 21.4M 0.0k nvme8n19 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 155.70 2.2M 2.3M 0.0k 21.9M 22.8M 0.0k nvme8n2 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 152.70 2.2M 2.2M 0.0k 21.8M 21.9M 0.0k nvme8n20 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 163.40 2.3M 2.4M 0.0k 22.9M 24.5M 0.0k nvme8n3 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 156.60 2.3M 2.2M 0.0k 22.7M 22.1M 0.0k nvme8n4 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 166.30 2.4M 2.4M 0.0k 23.8M 24.0M 0.0k nvme8n5 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 167.60 2.5M 2.4M 0.0k 24.7M 23.6M 0.0k nvme8n6 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 154.80 2.2M 2.2M 0.0k 21.7M 22.4M 0.0k nvme8n7 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 158.40 2.3M 2.3M 0.0k 23.2M 22.9M 0.0k nvme8n8 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 156.90 2.4M 2.2M 0.0k 23.7M 22.3M 0.0k nvme8n9 2026-02-24T16:49:36.113 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:37.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:37 trial151 ceph-mon[39453]: pgmap v297: 33 pgs: 33 active+clean; 57 GiB data, 24 GiB used, 2.7 TiB / 2.7 TiB avail; 146 MiB/s rd, 173 MiB/s wr, 11.90k op/s 2026-02-24T16:49:37.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:37 trial183 ceph-mon[43553]: pgmap v297: 33 pgs: 33 active+clean; 57 GiB data, 24 GiB used, 2.7 TiB / 2.7 TiB avail; 146 MiB/s rd, 173 MiB/s wr, 11.90k op/s 2026-02-24T16:49:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:37 trial153 ceph-mon[44017]: pgmap v297: 33 pgs: 33 active+clean; 57 GiB data, 24 GiB used, 2.7 TiB / 2.7 TiB avail; 146 MiB/s rd, 173 MiB/s wr, 11.90k op/s 2026-02-24T16:49:39.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:39 trial151 ceph-mon[39453]: pgmap v298: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 129 MiB/s wr, 8.95k op/s 2026-02-24T16:49:39.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:39 trial183 ceph-mon[43553]: pgmap v298: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 129 MiB/s wr, 8.95k op/s 2026-02-24T16:49:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:39 trial153 ceph-mon[44017]: pgmap v298: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 110 MiB/s rd, 129 MiB/s wr, 8.95k op/s 2026-02-24T16:49:40.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/766324137' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3518992765' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4235499840' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:40 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1919818798' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/766324137' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3518992765' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4235499840' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:40.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:40 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1919818798' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/766324137' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3518992765' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4235499840' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:41.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:40 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1919818798' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:41.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:41 trial151 ceph-mon[39453]: pgmap v299: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 109 MiB/s rd, 127 MiB/s wr, 8.87k op/s 2026-02-24T16:49:41.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:41 trial183 ceph-mon[43553]: pgmap v299: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 109 MiB/s rd, 127 MiB/s wr, 8.87k op/s 2026-02-24T16:49:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:41 trial153 ceph-mon[44017]: pgmap v299: 33 pgs: 33 active+clean; 57 GiB data, 25 GiB used, 2.7 TiB / 2.7 TiB avail; 109 MiB/s rd, 127 MiB/s wr, 8.87k op/s 2026-02-24T16:49:43.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:43 trial151 ceph-mon[39453]: pgmap v300: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 124 MiB/s rd, 145 MiB/s wr, 10.17k op/s 2026-02-24T16:49:43.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:43 trial183 ceph-mon[43553]: pgmap v300: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 124 MiB/s rd, 145 MiB/s wr, 10.17k op/s 2026-02-24T16:49:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:43 trial153 ceph-mon[44017]: pgmap v300: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 124 MiB/s rd, 145 MiB/s wr, 10.17k op/s 2026-02-24T16:49:45.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:45 trial151 ceph-mon[39453]: pgmap v301: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 102 MiB/s wr, 7.17k op/s 2026-02-24T16:49:45.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:45 trial183 ceph-mon[43553]: pgmap v301: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 102 MiB/s wr, 7.17k op/s 2026-02-24T16:49:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:45 trial153 ceph-mon[44017]: pgmap v301: 33 pgs: 33 active+clean; 57 GiB data, 27 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 102 MiB/s wr, 7.17k op/s 2026-02-24T16:49:46.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.60 1.6M 1.9M 0.0k 16.4M 18.5M 0.0k nvme12n1 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 133.30 1.7M 1.9M 0.0k 17.1M 19.3M 0.0k nvme12n10 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 140.40 2.0M 1.8M 0.0k 19.5M 18.0M 0.0k nvme12n11 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 139.50 2.0M 1.9M 0.0k 19.8M 19.3M 0.0k nvme12n12 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 128.10 1.9M 2.0M 0.0k 19.0M 19.9M 0.0k nvme12n13 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 134.50 2.0M 1.8M 0.0k 20.0M 17.6M 0.0k nvme12n14 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 131.00 1.8M 1.9M 0.0k 17.8M 18.5M 0.0k nvme12n15 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 127.30 1.9M 1.8M 0.0k 18.8M 17.6M 0.0k nvme12n16 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 129.90 1.8M 1.9M 0.0k 18.2M 19.4M 0.0k nvme12n17 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 128.40 1.8M 1.9M 0.0k 18.4M 18.8M 0.0k nvme12n18 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 125.20 1.7M 1.7M 0.0k 17.3M 16.5M 0.0k nvme12n19 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 135.20 1.9M 1.9M 0.0k 19.2M 19.1M 0.0k nvme12n2 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 124.20 1.6M 1.8M 0.0k 16.4M 17.7M 0.0k nvme12n20 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 143.20 2.1M 2.0M 0.0k 20.7M 20.3M 0.0k nvme12n3 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 135.50 1.9M 1.8M 0.0k 18.9M 18.4M 0.0k nvme12n4 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 123.10 1.7M 1.9M 0.0k 17.1M 19.0M 0.0k nvme12n5 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 131.00 1.8M 1.8M 0.0k 18.5M 18.2M 0.0k nvme12n6 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 126.40 1.7M 1.9M 0.0k 16.8M 18.8M 0.0k nvme12n7 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 132.90 1.9M 1.8M 0.0k 19.2M 18.4M 0.0k nvme12n8 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 133.60 2.0M 1.9M 0.0k 19.6M 19.2M 0.0k nvme12n9 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 136.30 2.0M 1.9M 0.0k 19.7M 19.3M 0.0k nvme4n1 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 126.90 1.7M 1.8M 0.0k 17.3M 18.0M 0.0k nvme4n10 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 132.00 1.8M 1.9M 0.0k 18.4M 19.3M 0.0k nvme4n11 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 127.30 1.9M 1.7M 0.0k 18.6M 17.2M 0.0k nvme4n12 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 125.10 1.8M 1.7M 0.0k 17.7M 17.2M 0.0k nvme4n13 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 135.10 1.9M 1.9M 0.0k 19.4M 19.2M 0.0k nvme4n14 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 132.60 1.8M 1.9M 0.0k 17.9M 19.2M 0.0k nvme4n15 2026-02-24T16:49:46.118 INFO:tasks.workunit.client.0.trial051.stdout: 126.80 1.8M 1.8M 0.0k 18.4M 18.2M 0.0k nvme4n16 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 129.00 1.8M 1.9M 0.0k 17.7M 18.7M 0.0k nvme4n17 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 135.10 1.9M 2.0M 0.0k 18.7M 20.3M 0.0k nvme4n18 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 133.90 1.9M 2.0M 0.0k 18.9M 19.8M 0.0k nvme4n19 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 129.60 1.9M 1.7M 0.0k 19.0M 16.9M 0.0k nvme4n2 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 138.80 1.9M 2.0M 0.0k 19.5M 19.9M 0.0k nvme4n20 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 127.60 1.9M 1.8M 0.0k 18.7M 17.9M 0.0k nvme4n3 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 129.40 1.8M 1.8M 0.0k 18.3M 17.7M 0.0k nvme4n4 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.50 1.8M 1.8M 0.0k 18.5M 18.4M 0.0k nvme4n5 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 135.50 2.0M 1.9M 0.0k 19.7M 19.3M 0.0k nvme4n6 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 142.50 1.9M 2.0M 0.0k 19.1M 19.5M 0.0k nvme4n7 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 133.70 1.9M 1.9M 0.0k 19.1M 18.9M 0.0k nvme4n8 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 132.60 1.9M 1.8M 0.0k 18.6M 18.5M 0.0k nvme4n9 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.80 2.0M 1.9M 0.0k 19.7M 19.0M 0.0k nvme8n1 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 130.90 1.9M 1.9M 0.0k 18.6M 19.5M 0.0k nvme8n10 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 129.20 1.7M 1.9M 0.0k 16.9M 18.5M 0.0k nvme8n11 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.60 1.9M 1.9M 0.0k 18.6M 19.1M 0.0k nvme8n12 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 128.60 1.9M 1.9M 0.0k 18.9M 18.8M 0.0k nvme8n13 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.00 1.9M 1.9M 0.0k 18.9M 19.0M 0.0k nvme8n14 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 127.00 1.8M 1.7M 0.0k 18.4M 17.1M 0.0k nvme8n15 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 140.20 2.0M 1.9M 0.0k 19.9M 19.5M 0.0k nvme8n16 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 131.00 1.8M 1.8M 0.0k 18.2M 18.4M 0.0k nvme8n17 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 128.00 1.9M 1.8M 0.0k 18.6M 18.1M 0.0k nvme8n18 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 128.10 1.9M 1.8M 0.0k 19.2M 17.6M 0.0k nvme8n19 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 131.60 2.0M 1.8M 0.0k 19.8M 18.4M 0.0k nvme8n2 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 128.60 1.8M 1.9M 0.0k 18.0M 19.3M 0.0k nvme8n20 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 138.00 1.9M 1.9M 0.0k 19.1M 19.1M 0.0k nvme8n3 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 135.80 1.8M 2.0M 0.0k 18.3M 19.8M 0.0k nvme8n4 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 125.10 1.6M 1.9M 0.0k 16.5M 18.8M 0.0k nvme8n5 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 139.40 2.0M 1.9M 0.0k 19.5M 19.0M 0.0k nvme8n6 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 132.40 1.9M 1.8M 0.0k 18.8M 18.2M 0.0k nvme8n7 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.50 1.9M 1.9M 0.0k 19.1M 19.4M 0.0k nvme8n8 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 134.10 1.9M 2.0M 0.0k 18.6M 19.6M 0.0k nvme8n9 2026-02-24T16:49:46.119 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:47 trial151 ceph-mon[39453]: pgmap v302: 33 pgs: 33 active+clean; 57 GiB data, 28 GiB used, 2.7 TiB / 2.7 TiB avail; 117 MiB/s rd, 137 MiB/s wr, 9.67k op/s 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3210394806' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2321254770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2685655530' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:47 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4196396183' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:47 trial183 ceph-mon[43553]: pgmap v302: 33 pgs: 33 active+clean; 57 GiB data, 28 GiB used, 2.7 TiB / 2.7 TiB avail; 117 MiB/s rd, 137 MiB/s wr, 9.67k op/s 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3210394806' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2321254770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2685655530' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:47.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:47 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4196396183' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:47 trial153 ceph-mon[44017]: pgmap v302: 33 pgs: 33 active+clean; 57 GiB data, 28 GiB used, 2.7 TiB / 2.7 TiB avail; 117 MiB/s rd, 137 MiB/s wr, 9.67k op/s 2026-02-24T16:49:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3210394806' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2321254770' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2685655530' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:47 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4196396183' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:49.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:49 trial151 ceph-mon[39453]: pgmap v303: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 105 MiB/s wr, 7.43k op/s 2026-02-24T16:49:49.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:49 trial183 ceph-mon[43553]: pgmap v303: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 105 MiB/s wr, 7.43k op/s 2026-02-24T16:49:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:49 trial153 ceph-mon[44017]: pgmap v303: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 105 MiB/s wr, 7.43k op/s 2026-02-24T16:49:51.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:51 trial151 ceph-mon[39453]: pgmap v304: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.63k op/s 2026-02-24T16:49:51.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:51 trial183 ceph-mon[43553]: pgmap v304: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.63k op/s 2026-02-24T16:49:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:51 trial153 ceph-mon[44017]: pgmap v304: 33 pgs: 33 active+clean; 57 GiB data, 29 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.63k op/s 2026-02-24T16:49:53.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:53 trial151 ceph-mon[39453]: pgmap v305: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 105 MiB/s rd, 123 MiB/s wr, 8.75k op/s 2026-02-24T16:49:53.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:53 trial183 ceph-mon[43553]: pgmap v305: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 105 MiB/s rd, 123 MiB/s wr, 8.75k op/s 2026-02-24T16:49:54.009 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:53 trial153 ceph-mon[44017]: pgmap v305: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 105 MiB/s rd, 123 MiB/s wr, 8.75k op/s 2026-02-24T16:49:54.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2639900954' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2772073814' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2635460772' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3267790175' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.985 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2639900954' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2772073814' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2635460772' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3267790175' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:49:54.987 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:49:54.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:54.988 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2639900954' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2772073814' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2635460772' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3267790175' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:49:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:49:55.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:49:55.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:55 trial151 ceph-mon[39453]: pgmap v306: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 75 MiB/s rd, 87 MiB/s wr, 6.26k op/s 2026-02-24T16:49:55.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:55 trial183 ceph-mon[43553]: pgmap v306: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 75 MiB/s rd, 87 MiB/s wr, 6.26k op/s 2026-02-24T16:49:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:55 trial153 ceph-mon[44017]: pgmap v306: 33 pgs: 33 active+clean; 57 GiB data, 30 GiB used, 2.7 TiB / 2.7 TiB avail; 75 MiB/s rd, 87 MiB/s wr, 6.26k op/s 2026-02-24T16:49:56.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:56.118 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 114.50 1.7M 1.6M 0.0k 16.6M 16.2M 0.0k nvme12n1 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 122.00 1.7M 1.8M 0.0k 17.1M 17.9M 0.0k nvme12n10 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 122.20 1.7M 1.6M 0.0k 17.2M 16.0M 0.0k nvme12n11 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.60 1.5M 1.7M 0.0k 14.9M 16.7M 0.0k nvme12n12 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 122.40 1.7M 1.7M 0.0k 17.4M 17.0M 0.0k nvme12n13 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 120.30 1.7M 1.8M 0.0k 17.4M 17.5M 0.0k nvme12n14 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 119.30 1.5M 1.7M 0.0k 15.4M 16.8M 0.0k nvme12n15 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 118.40 1.7M 1.6M 0.0k 17.0M 16.0M 0.0k nvme12n16 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 1.7M 1.7M 0.0k 16.8M 16.8M 0.0k nvme12n17 2026-02-24T16:49:56.119 INFO:tasks.workunit.client.0.trial051.stdout: 122.40 1.6M 1.6M 0.0k 16.2M 16.3M 0.0k nvme12n18 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 120.10 1.7M 1.6M 0.0k 17.0M 16.3M 0.0k nvme12n19 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 118.90 1.7M 1.6M 0.0k 16.7M 16.1M 0.0k nvme12n2 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 118.70 1.6M 1.6M 0.0k 16.2M 16.3M 0.0k nvme12n20 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 116.60 1.6M 1.7M 0.0k 15.6M 17.1M 0.0k nvme12n3 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.6M 1.7M 0.0k 15.7M 17.2M 0.0k nvme12n4 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 111.60 1.6M 1.6M 0.0k 15.8M 16.4M 0.0k nvme12n5 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 121.90 1.6M 1.6M 0.0k 15.7M 16.3M 0.0k nvme12n6 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 119.20 1.6M 1.6M 0.0k 16.0M 16.3M 0.0k nvme12n7 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 122.20 1.6M 1.7M 0.0k 16.5M 17.0M 0.0k nvme12n8 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 112.80 1.5M 1.6M 0.0k 15.2M 16.4M 0.0k nvme12n9 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 125.70 1.7M 1.7M 0.0k 17.3M 16.8M 0.0k nvme4n1 2026-02-24T16:49:56.120 INFO:tasks.workunit.client.0.trial051.stdout: 115.80 1.5M 1.7M 0.0k 15.4M 16.6M 0.0k nvme4n10 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.60 1.5M 1.7M 0.0k 15.2M 17.4M 0.0k nvme4n11 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 114.80 1.7M 1.5M 0.0k 17.1M 14.9M 0.0k nvme4n12 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.60 1.5M 1.7M 0.0k 15.4M 17.1M 0.0k nvme4n13 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 115.90 1.6M 1.6M 0.0k 15.7M 15.9M 0.0k nvme4n14 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.30 1.6M 1.6M 0.0k 15.8M 16.4M 0.0k nvme4n15 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.00 1.5M 1.7M 0.0k 15.4M 16.6M 0.0k nvme4n16 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.6M 1.6M 0.0k 15.9M 16.3M 0.0k nvme4n17 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 1.7M 1.7M 0.0k 16.5M 17.2M 0.0k nvme4n18 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 115.70 1.5M 1.6M 0.0k 15.2M 16.3M 0.0k nvme4n19 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 115.00 1.6M 1.6M 0.0k 16.0M 15.9M 0.0k nvme4n2 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.20 1.5M 1.6M 0.0k 14.9M 16.1M 0.0k nvme4n20 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.90 1.6M 1.8M 0.0k 16.1M 17.5M 0.0k nvme4n3 2026-02-24T16:49:56.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.7M 1.7M 0.0k 16.9M 16.9M 0.0k nvme4n4 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 114.90 1.5M 1.6M 0.0k 14.7M 16.4M 0.0k nvme4n5 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 119.20 1.6M 1.7M 0.0k 16.1M 16.8M 0.0k nvme4n6 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 123.30 1.8M 1.6M 0.0k 17.8M 15.6M 0.0k nvme4n7 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 117.50 1.6M 1.6M 0.0k 16.2M 15.9M 0.0k nvme4n8 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 111.40 1.6M 1.5M 0.0k 15.5M 15.1M 0.0k nvme4n9 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 123.60 1.7M 1.7M 0.0k 16.9M 16.8M 0.0k nvme8n1 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 122.70 1.7M 1.7M 0.0k 17.2M 16.5M 0.0k nvme8n10 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 125.50 1.7M 1.8M 0.0k 17.0M 18.2M 0.0k nvme8n11 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 126.30 1.7M 1.8M 0.0k 17.5M 17.5M 0.0k nvme8n12 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 127.50 1.8M 1.8M 0.0k 18.3M 17.5M 0.0k nvme8n13 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 124.00 1.6M 1.7M 0.0k 16.2M 16.9M 0.0k nvme8n14 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 114.40 1.6M 1.5M 0.0k 16.3M 15.2M 0.0k nvme8n15 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 125.50 1.7M 1.6M 0.0k 17.2M 16.4M 0.0k nvme8n16 2026-02-24T16:49:56.122 INFO:tasks.workunit.client.0.trial051.stdout: 122.60 1.7M 1.6M 0.0k 17.2M 15.8M 0.0k nvme8n17 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 115.40 1.5M 1.6M 0.0k 15.2M 15.9M 0.0k nvme8n18 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 118.70 1.7M 1.7M 0.0k 16.7M 17.0M 0.0k nvme8n19 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 122.50 1.7M 1.6M 0.0k 17.1M 16.1M 0.0k nvme8n2 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 121.40 1.7M 1.6M 0.0k 17.0M 16.2M 0.0k nvme8n20 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.80 1.6M 1.7M 0.0k 16.3M 17.1M 0.0k nvme8n3 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.7M 1.7M 0.0k 16.5M 16.6M 0.0k nvme8n4 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 118.00 1.7M 1.6M 0.0k 16.9M 15.5M 0.0k nvme8n5 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 121.70 1.7M 1.6M 0.0k 16.6M 16.2M 0.0k nvme8n6 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 113.90 1.5M 1.6M 0.0k 15.2M 16.2M 0.0k nvme8n7 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 125.20 1.9M 1.7M 0.0k 18.6M 17.1M 0.0k nvme8n8 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 118.80 1.6M 1.8M 0.0k 16.0M 17.5M 0.0k nvme8n9 2026-02-24T16:49:56.123 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:49:57.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:57 trial151 ceph-mon[39453]: pgmap v307: 33 pgs: 33 active+clean; 57 GiB data, 31 GiB used, 2.7 TiB / 2.7 TiB avail; 95 MiB/s rd, 110 MiB/s wr, 7.93k op/s 2026-02-24T16:49:57.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:57 trial183 ceph-mon[43553]: pgmap v307: 33 pgs: 33 active+clean; 57 GiB data, 31 GiB used, 2.7 TiB / 2.7 TiB avail; 95 MiB/s rd, 110 MiB/s wr, 7.93k op/s 2026-02-24T16:49:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:57 trial153 ceph-mon[44017]: pgmap v307: 33 pgs: 33 active+clean; 57 GiB data, 31 GiB used, 2.7 TiB / 2.7 TiB avail; 95 MiB/s rd, 110 MiB/s wr, 7.93k op/s 2026-02-24T16:49:59.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:49:59 trial151 ceph-mon[39453]: pgmap v308: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.71k op/s 2026-02-24T16:49:59.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:49:59 trial183 ceph-mon[43553]: pgmap v308: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.71k op/s 2026-02-24T16:50:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:49:59 trial153 ceph-mon[44017]: pgmap v308: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 93 MiB/s wr, 6.71k op/s 2026-02-24T16:50:00.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:00 trial151 ceph-mon[39453]: overall HEALTH_OK 2026-02-24T16:50:00.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:00 trial183 ceph-mon[43553]: overall HEALTH_OK 2026-02-24T16:50:01.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:00 trial153 ceph-mon[44017]: overall HEALTH_OK 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:01 trial151 ceph-mon[39453]: pgmap v309: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 83 MiB/s wr, 6.02k op/s 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1019837449' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3361357811' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/3789765617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:01 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/406145419' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:01 trial183 ceph-mon[43553]: pgmap v309: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 83 MiB/s wr, 6.02k op/s 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1019837449' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3361357811' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/3789765617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:01.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:01 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/406145419' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:01 trial153 ceph-mon[44017]: pgmap v309: 33 pgs: 33 active+clean; 58 GiB data, 32 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 83 MiB/s wr, 6.02k op/s 2026-02-24T16:50:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1019837449' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:02.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3361357811' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:02.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/3789765617' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:02.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:01 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/406145419' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:03.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:03 trial151 ceph-mon[39453]: pgmap v310: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 106 MiB/s rd, 123 MiB/s wr, 8.99k op/s 2026-02-24T16:50:03.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:03 trial183 ceph-mon[43553]: pgmap v310: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 106 MiB/s rd, 123 MiB/s wr, 8.99k op/s 2026-02-24T16:50:04.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:03 trial153 ceph-mon[44017]: pgmap v310: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 106 MiB/s rd, 123 MiB/s wr, 8.99k op/s 2026-02-24T16:50:05.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:05 trial151 ceph-mon[39453]: pgmap v311: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.93k op/s 2026-02-24T16:50:05.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:05 trial183 ceph-mon[43553]: pgmap v311: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.93k op/s 2026-02-24T16:50:06.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:05 trial153 ceph-mon[44017]: pgmap v311: 33 pgs: 33 active+clean; 58 GiB data, 33 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.93k op/s 2026-02-24T16:50:06.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.20 1.5M 1.5M 0.0k 15.1M 14.5M 0.0k nvme12n1 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 116.10 1.5M 1.6M 0.0k 15.3M 15.9M 0.0k nvme12n10 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 128.50 1.7M 1.6M 0.0k 17.4M 16.0M 0.0k nvme12n11 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 113.00 1.4M 1.6M 0.0k 14.3M 15.7M 0.0k nvme12n12 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.6M 1.7M 0.0k 15.9M 16.6M 0.0k nvme12n13 2026-02-24T16:50:06.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.50 1.6M 1.6M 0.0k 15.9M 15.6M 0.0k nvme12n14 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 118.40 1.6M 1.6M 0.0k 15.8M 16.2M 0.0k nvme12n15 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 117.30 1.6M 1.5M 0.0k 16.3M 15.0M 0.0k nvme12n16 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 113.60 1.5M 1.5M 0.0k 15.3M 15.4M 0.0k nvme12n17 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.5M 1.6M 0.0k 14.9M 16.2M 0.0k nvme12n18 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 111.40 1.4M 1.6M 0.0k 13.8M 15.6M 0.0k nvme12n19 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 114.30 1.7M 1.5M 0.0k 16.8M 15.4M 0.0k nvme12n2 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 1.6M 1.7M 0.0k 16.4M 16.5M 0.0k nvme12n20 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 128.20 1.8M 1.6M 0.0k 18.2M 15.6M 0.0k nvme12n3 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 114.30 1.5M 1.5M 0.0k 14.9M 15.4M 0.0k nvme12n4 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 114.50 1.6M 1.6M 0.0k 16.4M 16.0M 0.0k nvme12n5 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 115.90 1.5M 1.5M 0.0k 14.9M 15.3M 0.0k nvme12n6 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 122.40 1.6M 1.7M 0.0k 15.6M 17.5M 0.0k nvme12n7 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 114.80 1.6M 1.6M 0.0k 15.5M 15.9M 0.0k nvme12n8 2026-02-24T16:50:06.120 INFO:tasks.workunit.client.0.trial051.stdout: 117.70 1.5M 1.6M 0.0k 15.4M 16.2M 0.0k nvme12n9 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 121.20 1.5M 1.7M 0.0k 15.2M 17.2M 0.0k nvme4n1 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 109.70 1.4M 1.6M 0.0k 14.2M 15.9M 0.0k nvme4n10 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 1.7M 1.7M 0.0k 16.8M 16.7M 0.0k nvme4n11 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.70 1.5M 1.6M 0.0k 15.3M 16.3M 0.0k nvme4n12 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.20 1.7M 1.6M 0.0k 16.7M 15.8M 0.0k nvme4n13 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.70 1.7M 1.6M 0.0k 16.8M 15.5M 0.0k nvme4n14 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.70 1.5M 1.5M 0.0k 15.1M 15.5M 0.0k nvme4n15 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 110.70 1.4M 1.5M 0.0k 14.1M 15.0M 0.0k nvme4n16 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 122.70 1.7M 1.5M 0.0k 16.8M 15.3M 0.0k nvme4n17 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 111.60 1.5M 1.5M 0.0k 15.0M 15.4M 0.0k nvme4n18 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 123.00 1.6M 1.6M 0.0k 16.3M 16.5M 0.0k nvme4n19 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 111.20 1.4M 1.6M 0.0k 13.9M 15.7M 0.0k nvme4n2 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 120.00 1.7M 1.6M 0.0k 16.6M 15.8M 0.0k nvme4n20 2026-02-24T16:50:06.121 INFO:tasks.workunit.client.0.trial051.stdout: 120.70 1.7M 1.6M 0.0k 16.7M 16.4M 0.0k nvme4n3 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 119.60 1.8M 1.6M 0.0k 17.7M 16.3M 0.0k nvme4n4 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 121.30 1.6M 1.7M 0.0k 16.1M 16.6M 0.0k nvme4n5 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 116.00 1.6M 1.5M 0.0k 15.6M 15.4M 0.0k nvme4n6 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 1.6M 1.6M 0.0k 16.1M 16.0M 0.0k nvme4n7 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 108.40 1.3M 1.5M 0.0k 13.2M 15.0M 0.0k nvme4n8 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 118.40 1.7M 1.5M 0.0k 16.6M 15.5M 0.0k nvme4n9 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 117.20 1.5M 1.6M 0.0k 14.8M 15.9M 0.0k nvme8n1 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 116.40 1.6M 1.7M 0.0k 15.5M 16.7M 0.0k nvme8n10 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 119.10 1.7M 1.7M 0.0k 16.6M 17.0M 0.0k nvme8n11 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.7M 1.6M 0.0k 16.7M 15.7M 0.0k nvme8n12 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 119.50 1.7M 1.5M 0.0k 16.8M 15.2M 0.0k nvme8n13 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.6M 1.6M 0.0k 15.8M 15.7M 0.0k nvme8n14 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.7M 1.6M 0.0k 16.7M 15.7M 0.0k nvme8n15 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 110.80 1.4M 1.6M 0.0k 13.8M 15.6M 0.0k nvme8n16 2026-02-24T16:50:06.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.6M 1.5M 0.0k 15.7M 15.3M 0.0k nvme8n17 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.70 1.8M 1.6M 0.0k 17.7M 16.3M 0.0k nvme8n18 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 113.80 1.6M 1.5M 0.0k 16.1M 14.8M 0.0k nvme8n19 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 119.00 1.6M 1.5M 0.0k 16.5M 15.0M 0.0k nvme8n2 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 115.40 1.5M 1.5M 0.0k 15.3M 15.3M 0.0k nvme8n20 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 124.80 1.7M 1.8M 0.0k 16.9M 17.6M 0.0k nvme8n3 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 113.20 1.4M 1.6M 0.0k 14.3M 15.8M 0.0k nvme8n4 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.10 1.7M 1.5M 0.0k 17.0M 15.3M 0.0k nvme8n5 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.30 1.6M 1.6M 0.0k 16.2M 16.5M 0.0k nvme8n6 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 111.20 1.5M 1.6M 0.0k 14.9M 16.0M 0.0k nvme8n7 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 112.60 1.5M 1.6M 0.0k 15.0M 15.6M 0.0k nvme8n8 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 114.70 1.6M 1.6M 0.0k 15.7M 15.5M 0.0k nvme8n9 2026-02-24T16:50:06.123 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:07.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:07 trial151 ceph-mon[39453]: pgmap v312: 33 pgs: 33 active+clean; 58 GiB data, 34 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 102 MiB/s wr, 7.57k op/s 2026-02-24T16:50:07.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:07 trial183 ceph-mon[43553]: pgmap v312: 33 pgs: 33 active+clean; 58 GiB data, 34 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 102 MiB/s wr, 7.57k op/s 2026-02-24T16:50:08.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:07 trial153 ceph-mon[44017]: pgmap v312: 33 pgs: 33 active+clean; 58 GiB data, 34 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 102 MiB/s wr, 7.57k op/s 2026-02-24T16:50:08.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2212980865' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2959864172' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1860872352' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:08 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/110720931' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:08 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2212980865' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:08 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2959864172' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:08 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1860872352' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:08.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:08 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/110720931' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:09.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:08 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2212980865' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:09.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:08 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2959864172' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:09.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:08 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1860872352' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:09.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:08 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/110720931' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:09.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:09 trial151 ceph-mon[39453]: pgmap v313: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 97 MiB/s wr, 7.22k op/s 2026-02-24T16:50:09.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:09 trial183 ceph-mon[43553]: pgmap v313: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 97 MiB/s wr, 7.22k op/s 2026-02-24T16:50:10.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:09 trial153 ceph-mon[44017]: pgmap v313: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 97 MiB/s wr, 7.22k op/s 2026-02-24T16:50:11.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:11 trial151 ceph-mon[39453]: pgmap v314: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.92k op/s 2026-02-24T16:50:11.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:11 trial183 ceph-mon[43553]: pgmap v314: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.92k op/s 2026-02-24T16:50:12.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:11 trial153 ceph-mon[44017]: pgmap v314: 33 pgs: 33 active+clean; 58 GiB data, 35 GiB used, 2.7 TiB / 2.7 TiB avail; 70 MiB/s rd, 80 MiB/s wr, 5.92k op/s 2026-02-24T16:50:13.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:13 trial151 ceph-mon[39453]: pgmap v315: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 104 MiB/s rd, 119 MiB/s wr, 8.87k op/s 2026-02-24T16:50:13.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:13 trial183 ceph-mon[43553]: pgmap v315: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 104 MiB/s rd, 119 MiB/s wr, 8.87k op/s 2026-02-24T16:50:14.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:13 trial153 ceph-mon[44017]: pgmap v315: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 104 MiB/s rd, 119 MiB/s wr, 8.87k op/s 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:15 trial151 ceph-mon[39453]: pgmap v316: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 79 MiB/s wr, 5.90k op/s 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1939512743' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2865249598' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4082475670' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:15 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4008176660' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:15 trial183 ceph-mon[43553]: pgmap v316: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 79 MiB/s wr, 5.90k op/s 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1939512743' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2865249598' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4082475670' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:15.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:15 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4008176660' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:15 trial153 ceph-mon[44017]: pgmap v316: 33 pgs: 33 active+clean; 58 GiB data, 36 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 79 MiB/s wr, 5.90k op/s 2026-02-24T16:50:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1939512743' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2865249598' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4082475670' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:16.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:15 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4008176660' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:16.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:16.115 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 113.40 1.6M 1.6M 0.0k 15.7M 15.7M 0.0k nvme12n1 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 119.00 1.6M 1.5M 0.0k 16.0M 14.9M 0.0k nvme12n10 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 115.80 1.5M 1.5M 0.0k 14.9M 15.2M 0.0k nvme12n11 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 119.30 1.6M 1.6M 0.0k 15.6M 16.3M 0.0k nvme12n12 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 119.90 1.7M 1.6M 0.0k 16.5M 16.1M 0.0k nvme12n13 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 113.70 1.6M 1.5M 0.0k 16.4M 15.1M 0.0k nvme12n14 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 125.40 1.6M 1.7M 0.0k 16.1M 17.0M 0.0k nvme12n15 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 114.30 1.5M 1.5M 0.0k 14.6M 15.3M 0.0k nvme12n16 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 107.80 1.3M 1.5M 0.0k 13.0M 15.0M 0.0k nvme12n17 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.5M 1.5M 0.0k 15.2M 15.0M 0.0k nvme12n18 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 120.60 1.6M 1.5M 0.0k 16.3M 15.1M 0.0k nvme12n19 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 122.10 1.7M 1.5M 0.0k 16.9M 15.5M 0.0k nvme12n2 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 112.30 1.4M 1.5M 0.0k 13.8M 15.2M 0.0k nvme12n20 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 118.70 1.5M 1.6M 0.0k 15.2M 15.9M 0.0k nvme12n3 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 119.50 1.5M 1.6M 0.0k 15.0M 15.8M 0.0k nvme12n4 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 111.80 1.6M 1.5M 0.0k 15.5M 15.1M 0.0k nvme12n5 2026-02-24T16:50:16.116 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.6M 1.6M 0.0k 15.9M 16.2M 0.0k nvme12n6 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 111.30 1.4M 1.6M 0.0k 13.6M 15.7M 0.0k nvme12n7 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 122.80 1.7M 1.6M 0.0k 17.2M 15.7M 0.0k nvme12n8 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.7M 1.5M 0.0k 16.6M 15.5M 0.0k nvme12n9 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 120.50 1.6M 1.5M 0.0k 15.8M 15.4M 0.0k nvme4n1 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 119.30 1.6M 1.5M 0.0k 16.4M 15.1M 0.0k nvme4n10 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 123.00 1.7M 1.7M 0.0k 17.3M 16.7M 0.0k nvme4n11 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 121.10 1.6M 1.6M 0.0k 16.3M 16.1M 0.0k nvme4n12 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 122.50 1.8M 1.6M 0.0k 17.6M 15.6M 0.0k nvme4n13 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 120.80 1.6M 1.6M 0.0k 16.0M 15.7M 0.0k nvme4n14 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 124.50 1.6M 1.7M 0.0k 16.1M 16.6M 0.0k nvme4n15 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 113.10 1.6M 1.6M 0.0k 15.6M 15.6M 0.0k nvme4n16 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 118.50 1.6M 1.6M 0.0k 15.7M 16.2M 0.0k nvme4n17 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 117.30 1.6M 1.5M 0.0k 15.5M 15.2M 0.0k nvme4n18 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 122.30 1.5M 1.6M 0.0k 15.0M 15.9M 0.0k nvme4n19 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 121.40 1.7M 1.6M 0.0k 17.5M 16.1M 0.0k nvme4n2 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 118.40 1.6M 1.6M 0.0k 15.5M 16.1M 0.0k nvme4n20 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 120.20 1.6M 1.6M 0.0k 16.0M 16.0M 0.0k nvme4n3 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 109.50 1.4M 1.5M 0.0k 14.0M 14.9M 0.0k nvme4n4 2026-02-24T16:50:16.117 INFO:tasks.workunit.client.0.trial051.stdout: 117.20 1.6M 1.6M 0.0k 15.5M 15.6M 0.0k nvme4n5 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 111.60 1.5M 1.5M 0.0k 15.2M 14.8M 0.0k nvme4n6 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 114.60 1.5M 1.5M 0.0k 15.4M 15.5M 0.0k nvme4n7 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 109.80 1.5M 1.5M 0.0k 14.8M 15.4M 0.0k nvme4n8 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 111.80 1.5M 1.4M 0.0k 14.9M 14.2M 0.0k nvme4n9 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 121.20 1.8M 1.6M 0.0k 18.3M 16.3M 0.0k nvme8n1 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 119.30 1.6M 1.5M 0.0k 15.6M 15.1M 0.0k nvme8n10 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 119.10 1.6M 1.7M 0.0k 16.0M 17.3M 0.0k nvme8n11 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 117.50 1.5M 1.7M 0.0k 15.4M 16.9M 0.0k nvme8n12 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 111.90 1.4M 1.6M 0.0k 13.7M 15.7M 0.0k nvme8n13 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.6M 1.6M 0.0k 16.3M 15.9M 0.0k nvme8n14 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 119.30 1.6M 1.5M 0.0k 16.4M 15.2M 0.0k nvme8n15 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.5M 1.4M 0.0k 15.3M 14.4M 0.0k nvme8n16 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.20 1.5M 1.5M 0.0k 14.9M 14.6M 0.0k nvme8n17 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 112.40 1.5M 1.5M 0.0k 14.6M 15.0M 0.0k nvme8n18 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 112.30 1.5M 1.5M 0.0k 15.3M 14.9M 0.0k nvme8n19 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 112.10 1.4M 1.6M 0.0k 14.1M 15.6M 0.0k nvme8n2 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 114.50 1.5M 1.5M 0.0k 15.4M 15.3M 0.0k nvme8n20 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 119.20 1.5M 1.6M 0.0k 15.3M 15.6M 0.0k nvme8n3 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.7M 1.5M 0.0k 17.1M 14.8M 0.0k nvme8n4 2026-02-24T16:50:16.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.00 1.6M 1.5M 0.0k 16.3M 14.9M 0.0k nvme8n5 2026-02-24T16:50:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 123.10 1.6M 1.6M 0.0k 15.9M 16.2M 0.0k nvme8n6 2026-02-24T16:50:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.40 1.4M 1.6M 0.0k 13.6M 15.5M 0.0k nvme8n7 2026-02-24T16:50:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 113.00 1.5M 1.5M 0.0k 14.7M 14.6M 0.0k nvme8n8 2026-02-24T16:50:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.80 1.4M 1.6M 0.0k 14.5M 16.1M 0.0k nvme8n9 2026-02-24T16:50:16.119 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:17.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:17 trial151 ceph-mon[39453]: pgmap v317: 33 pgs: 33 active+clean; 58 GiB data, 37 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 101 MiB/s wr, 7.57k op/s 2026-02-24T16:50:17.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:17 trial183 ceph-mon[43553]: pgmap v317: 33 pgs: 33 active+clean; 58 GiB data, 37 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 101 MiB/s wr, 7.57k op/s 2026-02-24T16:50:18.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:17 trial153 ceph-mon[44017]: pgmap v317: 33 pgs: 33 active+clean; 58 GiB data, 37 GiB used, 2.7 TiB / 2.7 TiB avail; 89 MiB/s rd, 101 MiB/s wr, 7.57k op/s 2026-02-24T16:50:19.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:19 trial151 ceph-mon[39453]: pgmap v318: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 96 MiB/s wr, 7.22k op/s 2026-02-24T16:50:19.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:19 trial183 ceph-mon[43553]: pgmap v318: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 96 MiB/s wr, 7.22k op/s 2026-02-24T16:50:20.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:19 trial153 ceph-mon[44017]: pgmap v318: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 96 MiB/s wr, 7.22k op/s 2026-02-24T16:50:21.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:21 trial151 ceph-mon[39453]: pgmap v319: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 78 MiB/s wr, 5.91k op/s 2026-02-24T16:50:21.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:21 trial183 ceph-mon[43553]: pgmap v319: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 78 MiB/s wr, 5.91k op/s 2026-02-24T16:50:22.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:21 trial153 ceph-mon[44017]: pgmap v319: 33 pgs: 33 active+clean; 58 GiB data, 38 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 78 MiB/s wr, 5.91k op/s 2026-02-24T16:50:22.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3856568849' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/703341139' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1717464147' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:22 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/1492338930' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3856568849' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/703341139' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1717464147' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:22.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:22 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/1492338930' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:23.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3856568849' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:23.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/703341139' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:23.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1717464147' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:23.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:22 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/1492338930' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:23.734 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:23 trial183 ceph-mon[43553]: pgmap v320: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 103 MiB/s rd, 116 MiB/s wr, 8.84k op/s 2026-02-24T16:50:23.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:23 trial151 ceph-mon[39453]: pgmap v320: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 103 MiB/s rd, 116 MiB/s wr, 8.84k op/s 2026-02-24T16:50:24.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:23 trial153 ceph-mon[44017]: pgmap v320: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 103 MiB/s rd, 116 MiB/s wr, 8.84k op/s 2026-02-24T16:50:25.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:25 trial151 ceph-mon[39453]: pgmap v321: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 77 MiB/s wr, 5.89k op/s 2026-02-24T16:50:25.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:25 trial183 ceph-mon[43553]: pgmap v321: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 77 MiB/s wr, 5.89k op/s 2026-02-24T16:50:26.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:25 trial153 ceph-mon[44017]: pgmap v321: 33 pgs: 33 active+clean; 58 GiB data, 39 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 77 MiB/s wr, 5.89k op/s 2026-02-24T16:50:26.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 113.50 1.6M 1.5M 0.0k 16.0M 14.6M 0.0k nvme12n1 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 116.60 1.4M 1.6M 0.0k 14.2M 16.1M 0.0k nvme12n10 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 123.80 1.5M 1.7M 0.0k 15.3M 16.6M 0.0k nvme12n11 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 113.80 1.5M 1.5M 0.0k 15.5M 15.4M 0.0k nvme12n12 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.6M 1.5M 0.0k 15.7M 14.9M 0.0k nvme12n13 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 116.90 1.5M 1.5M 0.0k 14.5M 15.5M 0.0k nvme12n14 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 117.60 1.6M 1.4M 0.0k 15.8M 14.4M 0.0k nvme12n15 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 118.70 1.6M 1.5M 0.0k 15.9M 15.1M 0.0k nvme12n16 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 113.30 1.5M 1.5M 0.0k 15.3M 14.5M 0.0k nvme12n17 2026-02-24T16:50:26.111 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.5M 1.5M 0.0k 14.7M 15.5M 0.0k nvme12n18 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.60 1.5M 1.5M 0.0k 15.3M 15.3M 0.0k nvme12n19 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 116.90 1.5M 1.5M 0.0k 15.5M 14.5M 0.0k nvme12n2 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.70 1.5M 1.5M 0.0k 15.1M 14.7M 0.0k nvme12n20 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 119.60 1.5M 1.5M 0.0k 15.2M 15.4M 0.0k nvme12n3 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 128.20 1.7M 1.6M 0.0k 16.9M 15.9M 0.0k nvme12n4 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 108.00 1.4M 1.4M 0.0k 14.1M 14.4M 0.0k nvme12n5 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 123.30 1.6M 1.6M 0.0k 15.8M 15.7M 0.0k nvme12n6 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 121.30 1.7M 1.6M 0.0k 16.8M 15.6M 0.0k nvme12n7 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.30 1.6M 1.4M 0.0k 15.8M 14.2M 0.0k nvme12n8 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 120.60 1.6M 1.6M 0.0k 15.8M 16.0M 0.0k nvme12n9 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 119.10 1.5M 1.5M 0.0k 15.3M 15.3M 0.0k nvme4n1 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 118.80 1.6M 1.6M 0.0k 16.1M 15.9M 0.0k nvme4n10 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.20 1.5M 1.6M 0.0k 15.0M 16.0M 0.0k nvme4n11 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.7M 1.4M 0.0k 16.9M 13.9M 0.0k nvme4n12 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 108.90 1.3M 1.5M 0.0k 12.7M 15.0M 0.0k nvme4n13 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.60 1.5M 1.6M 0.0k 14.9M 15.7M 0.0k nvme4n14 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 112.70 1.4M 1.6M 0.0k 14.2M 15.9M 0.0k nvme4n15 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 118.80 1.5M 1.5M 0.0k 15.3M 14.6M 0.0k nvme4n16 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.90 1.5M 1.5M 0.0k 15.4M 15.2M 0.0k nvme4n17 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.90 1.5M 1.5M 0.0k 14.9M 15.4M 0.0k nvme4n18 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 120.10 1.6M 1.6M 0.0k 15.6M 16.4M 0.0k nvme4n19 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.50 1.6M 1.4M 0.0k 15.6M 13.9M 0.0k nvme4n2 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 113.30 1.5M 1.4M 0.0k 15.4M 14.2M 0.0k nvme4n20 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.70 1.6M 1.6M 0.0k 15.7M 15.9M 0.0k nvme4n3 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 113.10 1.6M 1.4M 0.0k 15.9M 13.9M 0.0k nvme4n4 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 112.90 1.4M 1.5M 0.0k 14.4M 15.2M 0.0k nvme4n5 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 119.10 1.6M 1.5M 0.0k 16.4M 15.0M 0.0k nvme4n6 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 1.4M 1.5M 0.0k 14.1M 15.1M 0.0k nvme4n7 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.5M 1.5M 0.0k 15.0M 15.2M 0.0k nvme4n8 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 117.70 1.5M 1.6M 0.0k 15.2M 16.2M 0.0k nvme4n9 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 114.70 1.5M 1.5M 0.0k 15.2M 15.1M 0.0k nvme8n1 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 113.80 1.5M 1.5M 0.0k 14.8M 15.3M 0.0k nvme8n10 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 119.20 1.6M 1.6M 0.0k 15.7M 15.6M 0.0k nvme8n11 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 125.20 1.7M 1.7M 0.0k 16.8M 16.7M 0.0k nvme8n12 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.5M 1.5M 0.0k 15.0M 15.3M 0.0k nvme8n13 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.6M 1.6M 0.0k 15.6M 16.2M 0.0k nvme8n14 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 105.40 1.3M 1.4M 0.0k 13.5M 13.5M 0.0k nvme8n15 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 113.30 1.5M 1.5M 0.0k 14.8M 15.3M 0.0k nvme8n16 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 108.50 1.4M 1.5M 0.0k 13.9M 14.9M 0.0k nvme8n17 2026-02-24T16:50:26.112 INFO:tasks.workunit.client.0.trial051.stdout: 110.90 1.5M 1.5M 0.0k 15.4M 14.8M 0.0k nvme8n18 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 107.00 1.4M 1.4M 0.0k 14.1M 14.2M 0.0k nvme8n19 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 118.00 1.5M 1.5M 0.0k 15.1M 14.6M 0.0k nvme8n2 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 113.90 1.5M 1.5M 0.0k 15.3M 14.6M 0.0k nvme8n20 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 121.80 1.7M 1.5M 0.0k 17.2M 15.1M 0.0k nvme8n3 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 114.40 1.5M 1.4M 0.0k 15.0M 14.0M 0.0k nvme8n4 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 111.10 1.5M 1.5M 0.0k 14.7M 14.9M 0.0k nvme8n5 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 117.00 1.4M 1.6M 0.0k 14.0M 15.8M 0.0k nvme8n6 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 114.80 1.5M 1.5M 0.0k 14.8M 15.3M 0.0k nvme8n7 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 116.30 1.6M 1.5M 0.0k 15.9M 15.2M 0.0k nvme8n8 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 113.10 1.5M 1.5M 0.0k 14.6M 15.1M 0.0k nvme8n9 2026-02-24T16:50:26.113 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:27.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:27 trial151 ceph-mon[39453]: pgmap v322: 33 pgs: 33 active+clean; 58 GiB data, 40 GiB used, 2.7 TiB / 2.7 TiB avail; 88 MiB/s rd, 98 MiB/s wr, 7.54k op/s 2026-02-24T16:50:27.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:27 trial183 ceph-mon[43553]: pgmap v322: 33 pgs: 33 active+clean; 58 GiB data, 40 GiB used, 2.7 TiB / 2.7 TiB avail; 88 MiB/s rd, 98 MiB/s wr, 7.54k op/s 2026-02-24T16:50:28.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:27 trial153 ceph-mon[44017]: pgmap v322: 33 pgs: 33 active+clean; 58 GiB data, 40 GiB used, 2.7 TiB / 2.7 TiB avail; 88 MiB/s rd, 98 MiB/s wr, 7.54k op/s 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:29 trial151 ceph-mon[39453]: pgmap v323: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 84 MiB/s rd, 93 MiB/s wr, 7.16k op/s 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4289831745' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2294787984' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/4263175720' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:29 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/2804224254' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:29 trial183 ceph-mon[43553]: pgmap v323: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 84 MiB/s rd, 93 MiB/s wr, 7.16k op/s 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4289831745' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2294787984' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/4263175720' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:29.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:29 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/2804224254' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:30.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:29 trial153 ceph-mon[44017]: pgmap v323: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 84 MiB/s rd, 93 MiB/s wr, 7.16k op/s 2026-02-24T16:50:30.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4289831745' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:30.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2294787984' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:30.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/4263175720' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:30.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:29 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/2804224254' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:31.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:31 trial151 ceph-mon[39453]: pgmap v324: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 76 MiB/s wr, 5.88k op/s 2026-02-24T16:50:31.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:31 trial183 ceph-mon[43553]: pgmap v324: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 76 MiB/s wr, 5.88k op/s 2026-02-24T16:50:32.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:31 trial153 ceph-mon[44017]: pgmap v324: 33 pgs: 33 active+clean; 58 GiB data, 41 GiB used, 2.7 TiB / 2.7 TiB avail; 69 MiB/s rd, 76 MiB/s wr, 5.88k op/s 2026-02-24T16:50:33.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:33 trial151 ceph-mon[39453]: pgmap v325: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 102 MiB/s rd, 114 MiB/s wr, 8.80k op/s 2026-02-24T16:50:33.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:33 trial183 ceph-mon[43553]: pgmap v325: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 102 MiB/s rd, 114 MiB/s wr, 8.80k op/s 2026-02-24T16:50:34.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:33 trial153 ceph-mon[44017]: pgmap v325: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 102 MiB/s rd, 114 MiB/s wr, 8.80k op/s 2026-02-24T16:50:35.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:35 trial151 ceph-mon[39453]: pgmap v326: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 68 MiB/s rd, 76 MiB/s wr, 5.87k op/s 2026-02-24T16:50:35.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:35 trial183 ceph-mon[43553]: pgmap v326: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 68 MiB/s rd, 76 MiB/s wr, 5.87k op/s 2026-02-24T16:50:36.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:35 trial153 ceph-mon[44017]: pgmap v326: 33 pgs: 33 active+clean; 58 GiB data, 42 GiB used, 2.7 TiB / 2.7 TiB avail; 68 MiB/s rd, 76 MiB/s wr, 5.87k op/s 2026-02-24T16:50:36.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 114.00 1.5M 1.5M 0.0k 14.6M 15.4M 0.0k nvme12n1 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 118.50 1.5M 1.5M 0.0k 15.4M 14.6M 0.0k nvme12n10 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 121.10 1.5M 1.6M 0.0k 15.1M 16.4M 0.0k nvme12n11 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 121.80 1.6M 1.5M 0.0k 16.0M 15.2M 0.0k nvme12n12 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 116.00 1.5M 1.5M 0.0k 14.6M 15.4M 0.0k nvme12n13 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.4M 1.5M 0.0k 14.4M 15.0M 0.0k nvme12n14 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 124.10 1.6M 1.5M 0.0k 16.1M 14.5M 0.0k nvme12n15 2026-02-24T16:50:36.116 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.5M 1.5M 0.0k 15.3M 15.4M 0.0k nvme12n16 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 114.50 1.5M 1.5M 0.0k 15.1M 14.6M 0.0k nvme12n17 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.7M 1.6M 0.0k 16.5M 15.5M 0.0k nvme12n18 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 107.60 1.3M 1.4M 0.0k 13.2M 14.2M 0.0k nvme12n19 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 109.40 1.3M 1.6M 0.0k 13.3M 15.5M 0.0k nvme12n2 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 114.90 1.4M 1.5M 0.0k 14.0M 14.7M 0.0k nvme12n20 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 116.70 1.5M 1.5M 0.0k 14.5M 15.4M 0.0k nvme12n3 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 112.30 1.4M 1.5M 0.0k 13.7M 15.5M 0.0k nvme12n4 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 107.80 1.4M 1.4M 0.0k 14.4M 14.2M 0.0k nvme12n5 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 110.10 1.3M 1.5M 0.0k 13.1M 15.3M 0.0k nvme12n6 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 114.80 1.4M 1.5M 0.0k 14.0M 15.2M 0.0k nvme12n7 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.5M 1.5M 0.0k 14.8M 15.0M 0.0k nvme12n8 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 117.90 1.6M 1.6M 0.0k 15.7M 16.0M 0.0k nvme12n9 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.5M 1.5M 0.0k 15.1M 14.9M 0.0k nvme4n1 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 115.70 1.5M 1.4M 0.0k 14.6M 14.2M 0.0k nvme4n10 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 122.70 1.6M 1.5M 0.0k 15.9M 15.2M 0.0k nvme4n11 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 112.10 1.5M 1.5M 0.0k 14.8M 14.9M 0.0k nvme4n12 2026-02-24T16:50:36.117 INFO:tasks.workunit.client.0.trial051.stdout: 115.80 1.5M 1.5M 0.0k 14.7M 15.4M 0.0k nvme4n13 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.00 1.5M 1.4M 0.0k 14.8M 14.1M 0.0k nvme4n14 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 121.20 1.6M 1.5M 0.0k 15.7M 14.7M 0.0k nvme4n15 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 116.00 1.6M 1.5M 0.0k 15.6M 14.6M 0.0k nvme4n16 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 121.10 1.5M 1.7M 0.0k 14.8M 16.6M 0.0k nvme4n17 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 113.80 1.3M 1.5M 0.0k 13.4M 14.7M 0.0k nvme4n18 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 121.70 1.6M 1.5M 0.0k 16.0M 15.0M 0.0k nvme4n19 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 113.70 1.5M 1.4M 0.0k 14.7M 14.1M 0.0k nvme4n2 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 114.80 1.4M 1.5M 0.0k 13.5M 14.9M 0.0k nvme4n20 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 108.70 1.3M 1.4M 0.0k 12.5M 14.4M 0.0k nvme4n3 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 110.40 1.3M 1.6M 0.0k 13.5M 15.7M 0.0k nvme4n4 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.60 1.5M 1.6M 0.0k 14.8M 15.8M 0.0k nvme4n5 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 111.10 1.4M 1.6M 0.0k 13.6M 15.9M 0.0k nvme4n6 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.60 1.4M 1.5M 0.0k 14.4M 15.3M 0.0k nvme4n7 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 112.70 1.4M 1.5M 0.0k 14.4M 14.6M 0.0k nvme4n8 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.80 1.5M 1.5M 0.0k 15.1M 15.2M 0.0k nvme4n9 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.4M 1.5M 0.0k 14.3M 14.5M 0.0k nvme8n1 2026-02-24T16:50:36.118 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.4M 1.5M 0.0k 14.4M 14.7M 0.0k nvme8n10 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 116.60 1.5M 1.5M 0.0k 14.5M 15.0M 0.0k nvme8n11 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.30 1.5M 1.6M 0.0k 14.7M 15.7M 0.0k nvme8n12 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 111.10 1.4M 1.5M 0.0k 14.3M 15.3M 0.0k nvme8n13 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 121.10 1.5M 1.5M 0.0k 15.2M 15.4M 0.0k nvme8n14 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 108.50 1.4M 1.5M 0.0k 13.9M 14.9M 0.0k nvme8n15 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 116.00 1.5M 1.5M 0.0k 14.7M 15.4M 0.0k nvme8n16 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 112.90 1.5M 1.4M 0.0k 14.8M 14.1M 0.0k nvme8n17 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 116.00 1.6M 1.4M 0.0k 15.8M 14.4M 0.0k nvme8n18 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 115.20 1.4M 1.5M 0.0k 14.3M 14.8M 0.0k nvme8n19 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 117.80 1.5M 1.5M 0.0k 15.4M 15.1M 0.0k nvme8n2 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 109.40 1.4M 1.4M 0.0k 14.4M 14.2M 0.0k nvme8n20 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 111.00 1.4M 1.4M 0.0k 14.2M 13.8M 0.0k nvme8n3 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.6M 1.6M 0.0k 16.3M 16.0M 0.0k nvme8n4 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 122.10 1.7M 1.6M 0.0k 16.7M 15.8M 0.0k nvme8n5 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 116.90 1.5M 1.5M 0.0k 14.6M 14.8M 0.0k nvme8n6 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 110.30 1.3M 1.4M 0.0k 13.4M 14.2M 0.0k nvme8n7 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 114.40 1.5M 1.5M 0.0k 14.7M 15.3M 0.0k nvme8n8 2026-02-24T16:50:36.119 INFO:tasks.workunit.client.0.trial051.stdout: 120.60 1.6M 1.4M 0.0k 15.8M 14.3M 0.0k nvme8n9 2026-02-24T16:50:36.120 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:36.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/4007773537' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/1517605672' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/2571312090' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.984 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:36 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/602280266' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/4007773537' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/1517605672' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/2571312090' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:36.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:36 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/602280266' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:37.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/4007773537' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:37.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/1517605672' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:37.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/2571312090' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:37.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:36 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/602280266' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:37.983 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:37 trial151 ceph-mon[39453]: pgmap v327: 33 pgs: 33 active+clean; 58 GiB data, 43 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 97 MiB/s wr, 7.52k op/s 2026-02-24T16:50:37.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:37 trial183 ceph-mon[43553]: pgmap v327: 33 pgs: 33 active+clean; 58 GiB data, 43 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 97 MiB/s wr, 7.52k op/s 2026-02-24T16:50:38.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:37 trial153 ceph-mon[44017]: pgmap v327: 33 pgs: 33 active+clean; 58 GiB data, 43 GiB used, 2.7 TiB / 2.7 TiB avail; 87 MiB/s rd, 97 MiB/s wr, 7.52k op/s 2026-02-24T16:50:39.983 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:39 trial183 ceph-mon[43553]: pgmap v328: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 92 MiB/s wr, 7.15k op/s 2026-02-24T16:50:40.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:39 trial153 ceph-mon[44017]: pgmap v328: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 92 MiB/s wr, 7.15k op/s 2026-02-24T16:50:40.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:39 trial151 ceph-mon[39453]: pgmap v328: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 92 MiB/s wr, 7.15k op/s 2026-02-24T16:50:42.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:41 trial153 ceph-mon[44017]: pgmap v329: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 75 MiB/s wr, 5.85k op/s 2026-02-24T16:50:42.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:41 trial151 ceph-mon[39453]: pgmap v329: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 75 MiB/s wr, 5.85k op/s 2026-02-24T16:50:42.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:41 trial183 ceph-mon[43553]: pgmap v329: 33 pgs: 33 active+clean; 58 GiB data, 44 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 75 MiB/s wr, 5.85k op/s 2026-02-24T16:50:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:43 trial153 ceph-mon[44017]: pgmap v330: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 101 MiB/s rd, 112 MiB/s wr, 8.83k op/s 2026-02-24T16:50:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/3133566983' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/721401789' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1583970258' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:43 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/4121874132' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:43 trial151 ceph-mon[39453]: pgmap v330: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 101 MiB/s rd, 112 MiB/s wr, 8.83k op/s 2026-02-24T16:50:44.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/3133566983' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/721401789' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1583970258' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:43 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/4121874132' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:43 trial183 ceph-mon[43553]: pgmap v330: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 101 MiB/s rd, 112 MiB/s wr, 8.83k op/s 2026-02-24T16:50:44.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/3133566983' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/721401789' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1583970258' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:44.236 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:43 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/4121874132' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:46.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:45 trial153 ceph-mon[44017]: pgmap v331: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 74 MiB/s wr, 5.90k op/s 2026-02-24T16:50:46.106 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:46.120 INFO:tasks.workunit.client.0.trial051.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2026-02-24T16:50:46.120 INFO:tasks.workunit.client.0.trial051.stdout: 115.20 1.4M 1.4M 0.0k 14.3M 13.8M 0.0k nvme12n1 2026-02-24T16:50:46.120 INFO:tasks.workunit.client.0.trial051.stdout: 118.30 1.5M 1.4M 0.0k 15.1M 14.1M 0.0k nvme12n10 2026-02-24T16:50:46.120 INFO:tasks.workunit.client.0.trial051.stdout: 124.40 1.5M 1.5M 0.0k 15.1M 15.2M 0.0k nvme12n11 2026-02-24T16:50:46.120 INFO:tasks.workunit.client.0.trial051.stdout: 113.30 1.4M 1.4M 0.0k 14.4M 13.7M 0.0k nvme12n12 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 125.40 1.6M 1.6M 0.0k 16.5M 15.9M 0.0k nvme12n13 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.20 1.3M 1.5M 0.0k 13.2M 14.6M 0.0k nvme12n14 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.4M 1.4M 0.0k 14.3M 14.5M 0.0k nvme12n15 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 1.5M 1.5M 0.0k 15.2M 15.1M 0.0k nvme12n16 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 121.00 1.7M 1.5M 0.0k 16.6M 14.8M 0.0k nvme12n17 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 119.10 1.5M 1.5M 0.0k 14.7M 14.7M 0.0k nvme12n18 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 111.40 1.4M 1.4M 0.0k 14.0M 13.9M 0.0k nvme12n19 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 117.80 1.5M 1.5M 0.0k 14.7M 14.9M 0.0k nvme12n2 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 111.50 1.4M 1.5M 0.0k 13.8M 15.0M 0.0k nvme12n20 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.10 1.4M 1.5M 0.0k 14.5M 14.7M 0.0k nvme12n3 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 118.40 1.4M 1.4M 0.0k 14.5M 13.6M 0.0k nvme12n4 2026-02-24T16:50:46.121 INFO:tasks.workunit.client.0.trial051.stdout: 112.70 1.4M 1.5M 0.0k 13.9M 14.6M 0.0k nvme12n5 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 118.20 1.5M 1.4M 0.0k 14.8M 14.3M 0.0k nvme12n6 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.10 1.4M 1.5M 0.0k 13.8M 15.0M 0.0k nvme12n7 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 117.40 1.6M 1.4M 0.0k 15.6M 14.3M 0.0k nvme12n8 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.90 1.5M 1.4M 0.0k 14.7M 14.3M 0.0k nvme12n9 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 131.00 1.7M 1.6M 0.0k 16.8M 15.7M 0.0k nvme4n1 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 117.70 1.4M 1.5M 0.0k 14.1M 15.2M 0.0k nvme4n10 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 123.60 1.8M 1.5M 0.0k 17.9M 14.5M 0.0k nvme4n11 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 106.70 1.4M 1.4M 0.0k 13.5M 13.6M 0.0k nvme4n12 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 114.70 1.4M 1.5M 0.0k 14.3M 14.9M 0.0k nvme4n13 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 115.20 1.4M 1.4M 0.0k 14.3M 14.2M 0.0k nvme4n14 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 125.60 1.6M 1.6M 0.0k 16.2M 16.3M 0.0k nvme4n15 2026-02-24T16:50:46.122 INFO:tasks.workunit.client.0.trial051.stdout: 114.30 1.5M 1.5M 0.0k 14.5M 14.7M 0.0k nvme4n16 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 114.60 1.5M 1.4M 0.0k 15.0M 14.1M 0.0k nvme4n17 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 112.80 1.4M 1.4M 0.0k 14.3M 14.2M 0.0k nvme4n18 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 128.10 1.6M 1.5M 0.0k 16.2M 15.3M 0.0k nvme4n19 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 113.70 1.5M 1.4M 0.0k 15.1M 14.0M 0.0k nvme4n2 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 119.90 1.5M 1.4M 0.0k 15.0M 14.3M 0.0k nvme4n20 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 114.30 1.4M 1.5M 0.0k 14.3M 14.9M 0.0k nvme4n3 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 115.40 1.5M 1.5M 0.0k 15.1M 15.1M 0.0k nvme4n4 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 112.50 1.5M 1.3M 0.0k 14.7M 13.1M 0.0k nvme4n5 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 114.40 1.4M 1.5M 0.0k 14.0M 14.6M 0.0k nvme4n6 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 110.40 1.3M 1.5M 0.0k 12.7M 14.7M 0.0k nvme4n7 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 115.60 1.4M 1.5M 0.0k 13.9M 15.0M 0.0k nvme4n8 2026-02-24T16:50:46.123 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.6M 1.5M 0.0k 15.6M 14.7M 0.0k nvme4n9 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 116.90 1.5M 1.4M 0.0k 14.5M 14.5M 0.0k nvme8n1 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 119.80 1.4M 1.5M 0.0k 14.0M 14.8M 0.0k nvme8n10 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 121.90 1.5M 1.5M 0.0k 15.2M 14.7M 0.0k nvme8n11 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 119.40 1.5M 1.5M 0.0k 14.9M 14.9M 0.0k nvme8n12 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 115.30 1.5M 1.5M 0.0k 14.7M 14.6M 0.0k nvme8n13 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 106.20 1.3M 1.4M 0.0k 13.1M 13.7M 0.0k nvme8n14 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 119.60 1.5M 1.5M 0.0k 14.9M 15.1M 0.0k nvme8n15 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 122.60 1.6M 1.5M 0.0k 15.6M 15.5M 0.0k nvme8n16 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 111.70 1.4M 1.3M 0.0k 14.1M 13.1M 0.0k nvme8n17 2026-02-24T16:50:46.124 INFO:tasks.workunit.client.0.trial051.stdout: 113.00 1.4M 1.5M 0.0k 13.8M 15.4M 0.0k nvme8n18 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 111.50 1.5M 1.3M 0.0k 14.8M 13.4M 0.0k nvme8n19 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 116.70 1.5M 1.4M 0.0k 14.7M 14.0M 0.0k nvme8n2 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 115.90 1.5M 1.4M 0.0k 15.5M 14.0M 0.0k nvme8n20 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 119.20 1.5M 1.5M 0.0k 15.2M 14.8M 0.0k nvme8n3 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 115.50 1.5M 1.4M 0.0k 14.8M 14.2M 0.0k nvme8n4 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 115.80 1.4M 1.5M 0.0k 13.7M 15.5M 0.0k nvme8n5 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 121.40 1.5M 1.6M 0.0k 14.8M 15.6M 0.0k nvme8n6 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 120.40 1.6M 1.6M 0.0k 16.3M 15.5M 0.0k nvme8n7 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 113.20 1.5M 1.5M 0.0k 15.5M 15.0M 0.0k nvme8n8 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 122.90 1.5M 1.5M 0.0k 15.4M 14.6M 0.0k nvme8n9 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:46.125 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:50:46.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:45 trial151 ceph-mon[39453]: pgmap v331: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 74 MiB/s wr, 5.90k op/s 2026-02-24T16:50:46.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:45 trial183 ceph-mon[43553]: pgmap v331: 33 pgs: 33 active+clean; 58 GiB data, 45 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 74 MiB/s wr, 5.90k op/s 2026-02-24T16:50:48.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:47 trial153 ceph-mon[44017]: pgmap v332: 33 pgs: 33 active+clean; 59 GiB data, 46 GiB used, 2.7 TiB / 2.7 TiB avail; 86 MiB/s rd, 95 MiB/s wr, 7.55k op/s 2026-02-24T16:50:48.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:47 trial151 ceph-mon[39453]: pgmap v332: 33 pgs: 33 active+clean; 59 GiB data, 46 GiB used, 2.7 TiB / 2.7 TiB avail; 86 MiB/s rd, 95 MiB/s wr, 7.55k op/s 2026-02-24T16:50:48.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:47 trial183 ceph-mon[43553]: pgmap v332: 33 pgs: 33 active+clean; 59 GiB data, 46 GiB used, 2.7 TiB / 2.7 TiB avail; 86 MiB/s rd, 95 MiB/s wr, 7.55k op/s 2026-02-24T16:50:50.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:49 trial153 ceph-mon[44017]: pgmap v333: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 90 MiB/s wr, 7.19k op/s 2026-02-24T16:50:50.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:49 trial151 ceph-mon[39453]: pgmap v333: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 90 MiB/s wr, 7.19k op/s 2026-02-24T16:50:50.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:49 trial183 ceph-mon[43553]: pgmap v333: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 82 MiB/s rd, 90 MiB/s wr, 7.19k op/s 2026-02-24T16:50:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/739760860' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/3318886280' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/968567297' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:50 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/946291544' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/739760860' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/3318886280' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/968567297' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:50 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/946291544' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/739760860' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/3318886280' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/968567297' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:51.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:50 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/946291544' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:52.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:51 trial153 ceph-mon[44017]: pgmap v334: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 73 MiB/s wr, 5.91k op/s 2026-02-24T16:50:52.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:51 trial151 ceph-mon[39453]: pgmap v334: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 73 MiB/s wr, 5.91k op/s 2026-02-24T16:50:52.233 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:51 trial183 ceph-mon[43553]: pgmap v334: 33 pgs: 33 active+clean; 59 GiB data, 47 GiB used, 2.7 TiB / 2.7 TiB avail; 67 MiB/s rd, 73 MiB/s wr, 5.91k op/s 2026-02-24T16:50:54.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:53 trial153 ceph-mon[44017]: pgmap v335: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 93 MiB/s rd, 101 MiB/s wr, 8.19k op/s 2026-02-24T16:50:54.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:53 trial151 ceph-mon[39453]: pgmap v335: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 93 MiB/s rd, 101 MiB/s wr, 8.19k op/s 2026-02-24T16:50:54.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:53 trial183 ceph-mon[43553]: pgmap v335: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 93 MiB/s rd, 101 MiB/s wr, 8.19k op/s 2026-02-24T16:50:55.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:54 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:50:55.161 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:54 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:50:55.165 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:54 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:50:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: pgmap v336: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 73 MiB/s wr, 5.86k op/s 2026-02-24T16:50:56.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.065 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.066 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial052", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial051", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.067 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:50:56.068 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.069 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:55 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:50:56.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: pgmap v336: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 73 MiB/s wr, 5.86k op/s 2026-02-24T16:50:56.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.235 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial052", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial051", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.236 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:50:56.237 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:50:56.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.238 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:55 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:50:56.238 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: pgmap v336: 33 pgs: 33 active+clean; 59 GiB data, 48 GiB used, 2.7 TiB / 2.7 TiB avail; 66 MiB/s rd, 73 MiB/s wr, 5.86k op/s 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.239 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial052", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial051", "name": "osd_memory_target"} : dispatch 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.240 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:50:56.241 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:55 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:50:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:57 trial153 ceph-mon[44017]: pgmap v337: 33 pgs: 33 active+clean; 59 GiB data, 49 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 92 MiB/s wr, 7.47k op/s 2026-02-24T16:50:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/2415577857' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/2229722218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/344288651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:57 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/3252727896' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:57 trial151 ceph-mon[39453]: pgmap v337: 33 pgs: 33 active+clean; 59 GiB data, 49 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 92 MiB/s wr, 7.47k op/s 2026-02-24T16:50:58.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/2415577857' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/2229722218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/344288651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.234 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:57 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/3252727896' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:57 trial183 ceph-mon[43553]: pgmap v337: 33 pgs: 33 active+clean; 59 GiB data, 49 GiB used, 2.7 TiB / 2.7 TiB avail; 85 MiB/s rd, 92 MiB/s wr, 7.47k op/s 2026-02-24T16:50:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/2415577857' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/2229722218' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/344288651' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:50:58.235 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:57 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/3252727896' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:00.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:50:59 trial153 ceph-mon[44017]: pgmap v338: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 87 MiB/s wr, 7.06k op/s 2026-02-24T16:51:00.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:50:59 trial151 ceph-mon[39453]: pgmap v338: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 87 MiB/s wr, 7.06k op/s 2026-02-24T16:51:00.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:50:59 trial183 ceph-mon[43553]: pgmap v338: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 80 MiB/s rd, 87 MiB/s wr, 7.06k op/s 2026-02-24T16:51:02.035 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:01 trial151 ceph-mon[39453]: pgmap v339: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 65 MiB/s rd, 71 MiB/s wr, 5.78k op/s 2026-02-24T16:51:02.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:01 trial153 ceph-mon[44017]: pgmap v339: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 65 MiB/s rd, 71 MiB/s wr, 5.78k op/s 2026-02-24T16:51:02.234 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:01 trial183 ceph-mon[43553]: pgmap v339: 33 pgs: 33 active+clean; 59 GiB data, 50 GiB used, 2.7 TiB / 2.7 TiB avail; 65 MiB/s rd, 71 MiB/s wr, 5.78k op/s 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout:job-/dev/nvme8n1: (groupid=0, jobs=60): err= 0: pid=64854: Tue Feb 24 16:51:03 2026 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: read: IOPS=4627, BW=131MiB/s (137MB/s)(15.3GiB/120017msec) 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: clat (usec): min=59, max=138747, avg=3073.40, stdev=5912.87 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: lat (usec): min=59, max=138747, avg=3073.45, stdev=5912.87 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: clat percentiles (usec): 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: | 1.00th=[ 118], 5.00th=[ 281], 10.00th=[ 318], 20.00th=[ 363], 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: | 30.00th=[ 412], 40.00th=[ 494], 50.00th=[ 840], 60.00th=[ 1549], 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: | 70.00th=[ 2540], 80.00th=[ 4047], 90.00th=[ 7701], 95.00th=[12911], 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: | 99.00th=[30016], 99.50th=[38536], 99.90th=[57934], 99.95th=[65799], 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: | 99.99th=[89654] 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: bw ( KiB/s): min=36072, max=541240, per=100.00%, avg=134289.68, stdev=1511.10, samples=14340 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: iops : min= 1560, max=16002, avg=4633.69, stdev=43.15, samples=14340 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: write: IOPS=4632, BW=131MiB/s (137MB/s)(15.4GiB/120017msec); 0 zone resets 2026-02-24T16:51:03.625 INFO:tasks.workunit.client.0.trial051.stdout: clat (usec): min=877, max=133656, avg=9844.15, stdev=8000.26 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: lat (usec): min=886, max=133662, avg=9878.04, stdev=7999.38 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: clat percentiles (usec): 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: | 1.00th=[ 1418], 5.00th=[ 1975], 10.00th=[ 2573], 20.00th=[ 3785], 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: | 30.00th=[ 4948], 40.00th=[ 6390], 50.00th=[ 7898], 60.00th=[ 9634], 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: | 70.00th=[11600], 80.00th=[14353], 90.00th=[19006], 95.00th=[24249], 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: | 99.00th=[40109], 99.50th=[47449], 99.90th=[67634], 99.95th=[76022], 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: | 99.99th=[95945] 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: bw ( KiB/s): min=49240, max=506096, per=100.00%, avg=134417.05, stdev=1459.97, samples=14340 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: iops : min= 2170, max=15036, avg=4639.15, stdev=41.02, samples=14340 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: lat (usec) : 100=0.23%, 250=1.28%, 500=18.72%, 750=3.95%, 1000=2.08% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: lat (msec) : 2=8.83%, 4=15.69%, 10=26.72%, 20=16.90%, 50=5.29% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: lat (msec) : 100=0.30%, 250=0.01% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: cpu : usr=0.29%, sys=0.06%, ctx=1187101, majf=0, minf=16088 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: issued rwts: total=555342,556030,0,0 short=0,0,0,0 dropped=0,0,0,0 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: latency : target=0, window=0, percentile=100.00%, depth=1 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout:Run status group 0 (all jobs): 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=15.3GiB (16.5GB), run=120017-120017msec 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: WRITE: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=15.4GiB (16.5GB), run=120017-120017msec 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout:Disk stats (read/write): 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n1: ios=9338/9334, merge=0/0, ticks=27508/91849, in_queue=119357, util=94.36% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n1: ios=8673/8802, merge=0/0, ticks=29459/89922, in_queue=119381, util=94.57% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n1: ios=9545/9674, merge=0/0, ticks=27761/91587, in_queue=119348, util=94.69% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n2: ios=9366/9343, merge=0/0, ticks=27763/91594, in_queue=119357, util=94.71% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n2: ios=9132/9046, merge=0/0, ticks=28364/90972, in_queue=119336, util=94.76% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n2: ios=9262/8998, merge=0/0, ticks=29988/89415, in_queue=119403, util=94.87% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n3: ios=9570/9580, merge=0/0, ticks=28243/91132, in_queue=119375, util=94.94% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n3: ios=9553/9566, merge=0/0, ticks=27188/92165, in_queue=119353, util=95.07% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n3: ios=8735/9023, merge=0/0, ticks=27384/92019, in_queue=119403, util=95.24% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n4: ios=9511/9312, merge=0/0, ticks=28114/91255, in_queue=119369, util=95.38% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n4: ios=9638/9654, merge=0/0, ticks=27541/91810, in_queue=119351, util=95.42% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n4: ios=9001/9225, merge=0/0, ticks=27970/91405, in_queue=119375, util=95.47% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n5: ios=9388/9322, merge=0/0, ticks=28269/91120, in_queue=119389, util=95.55% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n5: ios=8571/8851, merge=0/0, ticks=29071/90369, in_queue=119440, util=95.72% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n5: ios=9306/9386, merge=0/0, ticks=28136/91248, in_queue=119384, util=95.80% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n6: ios=9507/9590, merge=0/0, ticks=26680/92660, in_queue=119340, util=95.92% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n6: ios=9231/9173, merge=0/0, ticks=28338/91008, in_queue=119346, util=95.96% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n6: ios=9193/9263, merge=0/0, ticks=27843/91526, in_queue=119369, util=95.99% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n7: ios=9067/9086, merge=0/0, ticks=28620/90766, in_queue=119386, util=96.07% 2026-02-24T16:51:03.626 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n7: ios=9162/9453, merge=0/0, ticks=26972/92389, in_queue=119361, util=96.16% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n7: ios=9681/9823, merge=0/0, ticks=26872/92463, in_queue=119335, util=96.23% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n8: ios=9065/9050, merge=0/0, ticks=28394/90958, in_queue=119352, util=96.39% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n8: ios=9560/9430, merge=0/0, ticks=28158/91182, in_queue=119340, util=96.40% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n8: ios=9043/9157, merge=0/0, ticks=27891/91541, in_queue=119432, util=96.66% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n9: ios=9459/9342, merge=0/0, ticks=29055/90319, in_queue=119374, util=96.65% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n9: ios=9051/9042, merge=0/0, ticks=27835/91563, in_queue=119398, util=96.78% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n9: ios=9462/9368, merge=0/0, ticks=28971/90423, in_queue=119394, util=96.85% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n10: ios=9304/9388, merge=0/0, ticks=27331/91993, in_queue=119324, util=96.91% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n10: ios=9396/9351, merge=0/0, ticks=27326/92044, in_queue=119370, util=97.00% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n10: ios=9525/9618, merge=0/0, ticks=27858/91491, in_queue=119349, util=96.99% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n11: ios=9203/9338, merge=0/0, ticks=28271/91091, in_queue=119362, util=97.03% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n11: ios=9704/9691, merge=0/0, ticks=27563/91794, in_queue=119357, util=97.17% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n11: ios=9458/9426, merge=0/0, ticks=28189/91138, in_queue=119327, util=97.23% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n12: ios=9408/9642, merge=0/0, ticks=27062/92250, in_queue=119312, util=97.32% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n12: ios=9424/9342, merge=0/0, ticks=27229/92140, in_queue=119369, util=97.40% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n12: ios=8714/8551, merge=0/0, ticks=31286/88104, in_queue=119390, util=97.58% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n13: ios=8876/9213, merge=0/0, ticks=27058/92326, in_queue=119384, util=97.59% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n13: ios=9019/9002, merge=0/0, ticks=27753/91655, in_queue=119408, util=97.73% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n13: ios=8862/9003, merge=0/0, ticks=28437/90971, in_queue=119408, util=97.80% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n14: ios=9040/9098, merge=0/0, ticks=27515/91810, in_queue=119325, util=97.83% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n14: ios=9131/9133, merge=0/0, ticks=28445/90938, in_queue=119383, util=97.88% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n14: ios=9121/9157, merge=0/0, ticks=29199/90166, in_queue=119365, util=98.03% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n15: ios=8758/8667, merge=0/0, ticks=31098/88318, in_queue=119416, util=98.24% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n15: ios=9626/9612, merge=0/0, ticks=28518/90841, in_queue=119359, util=98.30% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n15: ios=9376/9530, merge=0/0, ticks=27532/91837, in_queue=119369, util=98.36% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n16: ios=9578/9586, merge=0/0, ticks=27767/91600, in_queue=119367, util=98.43% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n16: ios=9403/9087, merge=0/0, ticks=29978/89399, in_queue=119377, util=98.45% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n16: ios=8935/8939, merge=0/0, ticks=29095/90303, in_queue=119398, util=98.60% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n17: ios=9172/9107, merge=0/0, ticks=28753/90654, in_queue=119407, util=98.75% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n17: ios=9146/9142, merge=0/0, ticks=29736/89630, in_queue=119366, util=98.84% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n17: ios=9522/9423, merge=0/0, ticks=28049/91345, in_queue=119394, util=98.87% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n18: ios=9108/9102, merge=0/0, ticks=29684/89650, in_queue=119334, util=98.87% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n18: ios=9205/9187, merge=0/0, ticks=28628/90745, in_queue=119373, util=99.08% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n18: ios=9019/9036, merge=0/0, ticks=28413/90908, in_queue=119321, util=99.16% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n19: ios=9180/8957, merge=0/0, ticks=30090/89283, in_queue=119373, util=99.34% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n19: ios=9124/8988, merge=0/0, ticks=30447/88942, in_queue=119389, util=99.42% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n19: ios=9507/9613, merge=0/0, ticks=27827/91534, in_queue=119361, util=99.42% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme8n20: ios=9043/8828, merge=0/0, ticks=30028/89372, in_queue=119400, util=99.57% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme12n20: ios=9102/9174, merge=0/0, ticks=29036/90364, in_queue=119400, util=99.62% 2026-02-24T16:51:03.627 INFO:tasks.workunit.client.0.trial051.stdout: nvme4n20: ios=9574/9489, merge=0/0, ticks=28092/91294, in_queue=119386, util=99.81% 2026-02-24T16:51:03.629 INFO:tasks.workunit.client.0.trial051.stderr:+ '[' 0 -ne 0 ']' 2026-02-24T16:51:03.629 INFO:tasks.workunit.client.0.trial051.stdout:[nvmeof.fio] fio test successful! 2026-02-24T16:51:03.629 INFO:tasks.workunit.client.0.trial051.stderr:+ echo '[nvmeof.fio] fio test successful!' 2026-02-24T16:51:03.629 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T16:51:03.629 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-24T16:51:03.698 INFO:tasks.workunit:Stopping ['nvmeof/fio_test.sh --rbd_iostat'] on client.0... 2026-02-24T16:51:03.698 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-24T16:51:04.035 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:03 trial183 ceph-mon[43553]: pgmap v340: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 90 MiB/s rd, 98 MiB/s wr, 8.03k op/s 2026-02-24T16:51:04.061 DEBUG:teuthology.parallel:result is None 2026-02-24T16:51:04.061 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:51:04.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:03 trial153 ceph-mon[44017]: pgmap v340: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 90 MiB/s rd, 98 MiB/s wr, 8.03k op/s 2026-02-24T16:51:04.086 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:51:04.087 DEBUG:teuthology.orchestra.run.trial051:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T16:51:04.142 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T16:51:04.142 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2026-02-24T16:51:04.181 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2026-02-24T16:51:04.181 DEBUG:teuthology.orchestra.run.trial052:> rmdir -- /home/ubuntu/cephtest/mnt.1 2026-02-24T16:51:04.233 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:03 trial151 ceph-mon[39453]: pgmap v340: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 90 MiB/s rd, 98 MiB/s wr, 8.03k op/s 2026-02-24T16:51:04.238 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.1/client.1 2026-02-24T16:51:04.238 DEBUG:teuthology.run_tasks:Unwinding manager nvmeof 2026-02-24T16:51:04.255 INFO:tasks.nvmeof:[nvmeof] Removing nvmeof service 2026-02-24T16:51:04.255 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls 2026-02-24T16:51:04.373 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:HOST ADDR LABELS STATUS 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial051 10.20.193.51 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial052 10.20.193.52 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial151 10.20.193.151 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial153 10.20.193.153 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial183 10.20.193.183 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:trial191 10.20.193.191 2026-02-24T16:51:04.598 INFO:teuthology.orchestra.run.trial151.stdout:6 hosts in cluster 2026-02-24T16:51:04.634 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch rm nvmeof.mypool.mygroup0 2026-02-24T16:51:04.748 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:51:04.861 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:04 trial151 ceph-mon[39453]: from='client.? 10.20.193.151:0/1710437204' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:04.862 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:04 trial151 ceph-mon[39453]: from='client.? 10.20.193.183:0/125049230' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:04.862 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:04 trial151 ceph-mon[39453]: from='client.? 10.20.193.153:0/1963494118' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:04.862 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:04 trial151 ceph-mon[39453]: from='client.? 10.20.193.191:0/892644093' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:04.980 INFO:teuthology.orchestra.run.trial151.stdout:Removed service nvmeof.mypool.mygroup0 2026-02-24T16:51:05.019 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph orch host ls 2026-02-24T16:51:05.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:04 trial153 ceph-mon[44017]: from='client.? 10.20.193.151:0/1710437204' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:04 trial153 ceph-mon[44017]: from='client.? 10.20.193.183:0/125049230' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:04 trial153 ceph-mon[44017]: from='client.? 10.20.193.153:0/1963494118' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.064 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:04 trial153 ceph-mon[44017]: from='client.? 10.20.193.191:0/892644093' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.137 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:51:05.187 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:04 trial183 ceph-mon[43553]: from='client.? 10.20.193.151:0/1710437204' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.187 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:04 trial183 ceph-mon[43553]: from='client.? 10.20.193.183:0/125049230' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.187 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:04 trial183 ceph-mon[43553]: from='client.? 10.20.193.153:0/1963494118' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.187 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:04 trial183 ceph-mon[43553]: from='client.? 10.20.193.191:0/892644093' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2026-02-24T16:51:05.358 INFO:teuthology.orchestra.run.trial151.stdout:HOST ADDR LABELS STATUS 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial051 10.20.193.51 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial052 10.20.193.52 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial151 10.20.193.151 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial153 10.20.193.153 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial183 10.20.193.183 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:trial191 10.20.193.191 2026-02-24T16:51:05.359 INFO:teuthology.orchestra.run.trial151.stdout:6 hosts in cluster 2026-02-24T16:51:05.397 INFO:tasks.nvmeof:[nvmeof] Nvmeof teardown completed! 2026-02-24T16:51:05.397 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2026-02-24T16:51:05.414 INFO:tasks.cephadm:Teardown begin 2026-02-24T16:51:05.415 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.442 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.470 DEBUG:teuthology.orchestra.run.trial151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.494 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.520 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.547 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:51:05.613 INFO:tasks.cephadm:Disabling cephadm mgr module 2026-02-24T16:51:05.613 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 -- ceph mgr module disable cephadm 2026-02-24T16:51:05.733 INFO:teuthology.orchestra.run.trial151.stderr:Inferring config /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/mon.a/config 2026-02-24T16:51:05.751 INFO:teuthology.orchestra.run.trial151.stderr:Error: statfs /etc/ceph/ceph.client.admin.keyring: no such file or directory 2026-02-24T16:51:05.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:05 trial151 systemd[1]: Stopping Ceph nvmeof.nvmeof.a for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:05.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:05] INFO server.py:42 (2): GatewayServer: SIGTERM received signum=15 2026-02-24T16:51:05.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:05] INFO server.py:197 (2): GatewayServer is terminating gracefully... 2026-02-24T16:51:05.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:05] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.a) pid 19 args ['/usr/bin/ceph-nvmeof-monitor-client', '--gateway-name', 'client.nvmeof.nvmeof.a', '--gateway-address', '10.20.193.151:5500', '--gateway-pool', 'mypool', '--gateway-group', 'mygroup0', '--monitor-group-address', '10.20.193.151:5499', '-c', '/etc/ceph/ceph.conf', '-n', 'client.nvmeof.nvmeof.a', '-k', '/etc/ceph/keyring'] ... 2026-02-24T16:51:05.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:05] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.a) pid 76 args ['/usr/local/bin/nvmf_tgt', '--wait-for-rpc', '-u', '-r', '/var/tmp/spdk.sock', '--lcores', '(0-3)'] ... 2026-02-24T16:51:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2026-02-24T16:51:05.772 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2026-02-24T16:51:05.772 DEBUG:teuthology.orchestra.run.trial051:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.789 DEBUG:teuthology.orchestra.run.trial052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.805 DEBUG:teuthology.orchestra.run.trial151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='client.37839 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: pgmap v341: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 77 MiB/s wr, 6.37k op/s 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.814 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.815 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:05 trial153 ceph-mon[44017]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:51:05.819 DEBUG:teuthology.orchestra.run.trial153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.833 DEBUG:teuthology.orchestra.run.trial183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.846 DEBUG:teuthology.orchestra.run.trial191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2026-02-24T16:51:05.859 INFO:tasks.cephadm:Stopping all daemons... 2026-02-24T16:51:05.860 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2026-02-24T16:51:05.860 DEBUG:teuthology.orchestra.run.trial151:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a 2026-02-24T16:51:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='client.37839 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:51:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: pgmap v341: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 77 MiB/s wr, 6.37k op/s 2026-02-24T16:51:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:51:05.984 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.985 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:51:05.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:51:05.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:05.986 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:05 trial183 ceph-mon[43553]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='client.37839 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: pgmap v341: 33 pgs: 33 active+clean; 59 GiB data, 51 GiB used, 2.7 TiB / 2.7 TiB avail; 71 MiB/s rd, 77 MiB/s wr, 6.37k op/s 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.006 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-mon[39453]: from='mgr.14152 10.20.193.151:0/3660693628' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 systemd[1]: Stopping Ceph mon.a for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a[39449]: 2026-02-24T16:51:05.958+0000 7f6211476640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a[39449]: 2026-02-24T16:51:05.958+0000 7f6211476640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:05 trial151 podman[114783]: 2026-02-24 16:51:05.999508099 +0000 UTC m=+0.053538880 container died 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260223, org.label-schema.license=GPLv2, io.buildah.version=1.33.7, org.label-schema.vendor=CentOS, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-24T16:51:06.007 INFO:journalctl@ceph.mon.a.trial151.stdout:Feb 24 16:51:06 trial151 podman[114783]: 2026-02-24 16:51:06.009807583 +0000 UTC m=+0.063838374 container remove 448ac3ea15d18fb21c4dd85b68ae557106e015e473ba1297015475180e8feea0 (image=quay.ceph.io/ceph-ci/ceph:618c318a7a0b287ab5aeff207b7bf1d73ef45044, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-a, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260223, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2026-02-24T16:51:06.079 DEBUG:teuthology.orchestra.run.trial151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.a.service' 2026-02-24T16:51:06.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:06.114 INFO:tasks.cephadm.mon.a:Stopped mon.a 2026-02-24T16:51:06.115 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2026-02-24T16:51:06.115 DEBUG:teuthology.orchestra.run.trial153:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.b 2026-02-24T16:51:06.387 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:06 trial153 systemd[1]: Stopping Ceph mon.b for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:06.387 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-b[44013]: 2026-02-24T16:51:06.219+0000 7fbbaf2a4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2026-02-24T16:51:06.387 INFO:journalctl@ceph.mon.b.trial153.stdout:Feb 24 16:51:06 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-b[44013]: 2026-02-24T16:51:06.219+0000 7fbbaf2a4640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2026-02-24T16:51:06.474 DEBUG:teuthology.orchestra.run.trial153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.b.service' 2026-02-24T16:51:06.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:06.512 INFO:tasks.cephadm.mon.c:Stopped mon.b 2026-02-24T16:51:06.512 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2026-02-24T16:51:06.512 DEBUG:teuthology.orchestra.run.trial183:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.c 2026-02-24T16:51:06.788 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:06 trial183 systemd[1]: Stopping Ceph mon.c for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:06.789 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-c[43549]: 2026-02-24T16:51:06.610+0000 7f49fd0af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2026-02-24T16:51:06.789 INFO:journalctl@ceph.mon.c.trial183.stdout:Feb 24 16:51:06 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-mon-c[43549]: 2026-02-24T16:51:06.610+0000 7f49fd0af640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2026-02-24T16:51:06.864 DEBUG:teuthology.orchestra.run.trial183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mon.c.service' 2026-02-24T16:51:06.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:06.904 INFO:tasks.cephadm.mon.c:Stopped mon.c 2026-02-24T16:51:06.904 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2026-02-24T16:51:06.904 DEBUG:teuthology.orchestra.run.trial151:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x 2026-02-24T16:51:07.107 DEBUG:teuthology.orchestra.run.trial151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@mgr.x.service' 2026-02-24T16:51:07.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:07.139 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2026-02-24T16:51:07.139 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2026-02-24T16:51:07.139 DEBUG:teuthology.orchestra.run.trial151:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.0 2026-02-24T16:51:07.455 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:07 trial151 systemd[1]: Stopping Ceph osd.0 for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:07.455 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0[57357]: 2026-02-24T16:51:07.276+0000 7f232c943640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-24T16:51:07.455 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0[57357]: 2026-02-24T16:51:07.276+0000 7f232c943640 -1 osd.0 32 *** Got signal Terminated *** 2026-02-24T16:51:07.455 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0[57357]: 2026-02-24T16:51:07.276+0000 7f232c943640 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO server.py:233 (2): Stopping the server... 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO server.py:835 (2): Terminating discovery service... 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO server.py:815 (2): Sending signal SIGINT to Discovery service process 81 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO state.py:1126 (81): Cleanup OMAP on exit (discovery-trial151) 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO server.py:837 (2): Discovery service terminated 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO state.py:1126 (2): Cleanup OMAP on exit (gateway-client.nvmeof.nvmeof.a) 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO server.py:245 (2): Exiting the gateway process. 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a[75036]: [24-Feb-2026 16:51:07] INFO utils.py:560 (2): Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof.a/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof.a/nvmeof-log.gz 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 podman[114651]: 2026-02-24 16:51:07.383847588 +0000 UTC m=+1.661338593 container died ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a, com.redhat.component=ubi9-container, io.spdk.build.makeflags=, summary=Ceph NVMe over Fabrics Gateway, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., name=ceph-nvmeof, io.ceph.version=1.7.1, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.nvmeof=, io.ceph.git.branch=devel, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.summary=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., distribution-scope=public, maintainer=Ceph Developers , io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.maintainer=Ceph Developers , io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.branch=undefined, url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.ceph-release=20.2.0, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, release=, vendor=Red Hat, Inc., architecture=x86_64, io.spdk.version=25.09, io.ceph.component=ceph-nvmeof, io.openshift.wants=ceph,rbd, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.29.0, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, build-date=2026-02-24 13:05:47 UTC, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, vcs-type=git, version=1.7.1, org.centos.version=9.0-30.el9) 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 podman[114651]: 2026-02-24 16:51:07.395266621 +0000 UTC m=+1.672757626 container remove ad0de735ebd5f3cb1058f5c686f205c17cf18f4be7cbe6e00027b06783968b21 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.version=1.7.1, io.spdk.build.makeflags=, io.openshift.wants=ceph,rbd, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vendor=Red Hat, Inc., maintainer=Ceph Developers , org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.spdk.git.branch=undefined, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.version=25.09, url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.branch=devel, io.spdk.build.ceph-release=20.2.0, io.spdk.build.target-arch=x86-64-v2, release=, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, org.centos.version=9.0-30.el9, name=ceph-nvmeof, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.build.pkgdep-args=--rbd, architecture=x86_64, distribution-scope=public, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, build-date=2026-02-24 13:05:47 UTC, vcs-type=git, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, com.redhat.component=ubi9-container, version=1.7.1, io.ceph.nvmeof=, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.maintainer=Ceph Developers , io.ceph.summary=Ceph NVMe over Fabrics Gateway) 2026-02-24T16:51:07.455 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 bash[114651]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-a 2026-02-24T16:51:07.733 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.a.service: Deactivated successfully. 2026-02-24T16:51:07.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 systemd[1]: Stopped Ceph nvmeof.nvmeof.a for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:51:07.734 INFO:journalctl@ceph.nvmeof.nvmeof.a.trial151.stdout:Feb 24 16:51:07 trial151 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.a.service: Consumed 33min 59.463s CPU time, 1.5G memory peak. 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115096]: 2026-02-24 16:51:12.508372924 +0000 UTC m=+5.243575514 container died 23fbc7cc9f712893fb5ad213d0ca3b7140c71a7823486651bd57fe83237a8ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, ceph=True, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260223, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115096]: 2026-02-24 16:51:12.518429574 +0000 UTC m=+5.253632163 container remove 23fbc7cc9f712893fb5ad213d0ca3b7140c71a7823486651bd57fe83237a8ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0, io.buildah.version=1.33.7, org.label-schema.schema-version=1.0, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, ceph=True, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260223) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 bash[115096]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.639949079 +0000 UTC m=+0.015346255 container create dfdb0ef3d61df59bc0d3e0db9f17a1ca9bd2f7937440ac05627d284c56dc1b5f (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0-deactivate, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, io.buildah.version=1.33.7, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260223, org.label-schema.license=GPLv2) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.67050522 +0000 UTC m=+0.045902406 container init dfdb0ef3d61df59bc0d3e0db9f17a1ca9bd2f7937440ac05627d284c56dc1b5f (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-yuri8-testing-2026-02-23-2128, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, io.buildah.version=1.33.7, ceph=True, org.label-schema.build-date=20260223, org.label-schema.license=GPLv2) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.673261306 +0000 UTC m=+0.048658492 container start dfdb0ef3d61df59bc0d3e0db9f17a1ca9bd2f7937440ac05627d284c56dc1b5f (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.7, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.673854498 +0000 UTC m=+0.049251684 container attach dfdb0ef3d61df59bc0d3e0db9f17a1ca9bd2f7937440ac05627d284c56dc1b5f (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0-deactivate, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.33.7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.633224344 +0000 UTC m=+0.008621530 image pull c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18 2026-02-24T16:51:12.792 INFO:journalctl@ceph.osd.0.trial151.stdout:Feb 24 16:51:12 trial151 podman[115325]: 2026-02-24 16:51:12.79551133 +0000 UTC m=+0.170908516 container died dfdb0ef3d61df59bc0d3e0db9f17a1ca9bd2f7937440ac05627d284c56dc1b5f (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-0-deactivate, org.label-schema.build-date=20260223, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.opencontainers.image.authors=Ceph Release Team , ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.33.7, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2) 2026-02-24T16:51:12.830 DEBUG:teuthology.orchestra.run.trial151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.0.service' 2026-02-24T16:51:12.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:12.865 INFO:tasks.cephadm.osd.0:Stopped osd.0 2026-02-24T16:51:12.865 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2026-02-24T16:51:12.865 DEBUG:teuthology.orchestra.run.trial153:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.1 2026-02-24T16:51:13.314 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:12 trial153 systemd[1]: Stopping Ceph osd.1 for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:13.314 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1[47523]: 2026-02-24T16:51:12.970+0000 7f9c628af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-24T16:51:13.314 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1[47523]: 2026-02-24T16:51:12.970+0000 7f9c628af640 -1 osd.1 32 *** Got signal Terminated *** 2026-02-24T16:51:13.314 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:12 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1[47523]: 2026-02-24T16:51:12.970+0000 7f9c628af640 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-24T16:51:18.535 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[71877]: 2026-02-24 16:51:18.222863687 +0000 UTC m=+5.264073832 container died dc0ca4deef6747be8c41125ccba7b0a81990623034b8eca9bf5df1eba67f0f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, io.buildah.version=1.33.7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[71877]: 2026-02-24 16:51:18.231424437 +0000 UTC m=+5.272634582 container remove dc0ca4deef6747be8c41125ccba7b0a81990623034b8eca9bf5df1eba67f0f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.build-date=20260223, OSD_FLAVOR=default) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 bash[71877]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[72025]: 2026-02-24 16:51:18.357680956 +0000 UTC m=+0.015486359 container create acc95ae505eecd093ccb29ba8cc83e83184ecd38b9f9cc019faef0ac5d3c5d0e (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1-deactivate, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, io.buildah.version=1.33.7, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[72025]: 2026-02-24 16:51:18.39098526 +0000 UTC m=+0.048790663 container init acc95ae505eecd093ccb29ba8cc83e83184ecd38b9f9cc019faef0ac5d3c5d0e (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1-deactivate, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260223, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[72025]: 2026-02-24 16:51:18.393886062 +0000 UTC m=+0.051691465 container start acc95ae505eecd093ccb29ba8cc83e83184ecd38b9f9cc019faef0ac5d3c5d0e (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1-deactivate, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.build-date=20260223, io.buildah.version=1.33.7) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[72025]: 2026-02-24 16:51:18.394549494 +0000 UTC m=+0.052354897 container attach acc95ae505eecd093ccb29ba8cc83e83184ecd38b9f9cc019faef0ac5d3c5d0e (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20260223, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-24T16:51:18.536 INFO:journalctl@ceph.osd.1.trial153.stdout:Feb 24 16:51:18 trial153 podman[72025]: 2026-02-24 16:51:18.350990338 +0000 UTC m=+0.008795741 image pull c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18 2026-02-24T16:51:18.567 DEBUG:teuthology.orchestra.run.trial153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.1.service' 2026-02-24T16:51:18.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:18.604 INFO:tasks.cephadm.osd.1:Stopped osd.1 2026-02-24T16:51:18.605 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-24T16:51:18.605 DEBUG:teuthology.orchestra.run.trial183:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.2 2026-02-24T16:51:18.984 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:18 trial183 systemd[1]: Stopping Ceph osd.2 for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:18.984 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:51:18.699+0000 7f066c040640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-24T16:51:18.984 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:51:18.699+0000 7f066c040640 -1 osd.2 32 *** Got signal Terminated *** 2026-02-24T16:51:18.985 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:18 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2[48623]: 2026-02-24T16:51:18.699+0000 7f066c040640 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:23 trial183 podman[75525]: 2026-02-24 16:51:23.962014335 +0000 UTC m=+5.273739027 container died a2d875db87ff655458449538c85ceeb988f5ebe93d5095a02348e70627526c40 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.license=GPLv2, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260223, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:23 trial183 podman[75525]: 2026-02-24 16:51:23.969961511 +0000 UTC m=+5.281686203 container remove a2d875db87ff655458449538c85ceeb988f5ebe93d5095a02348e70627526c40 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.build-date=20260223, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:23 trial183 bash[75525]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:24 trial183 podman[75671]: 2026-02-24 16:51:24.085843722 +0000 UTC m=+0.016006290 container create 69c1b82aab75577719e29424b4263597cf6295d26a6ca47bb85865f7207eca74 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260223, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.33.7, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:24 trial183 podman[75671]: 2026-02-24 16:51:24.119503363 +0000 UTC m=+0.049665951 container init 69c1b82aab75577719e29424b4263597cf6295d26a6ca47bb85865f7207eca74 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.vendor=CentOS, io.buildah.version=1.33.7, org.label-schema.build-date=20260223, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:24 trial183 podman[75671]: 2026-02-24 16:51:24.122384644 +0000 UTC m=+0.052547212 container start 69c1b82aab75577719e29424b4263597cf6295d26a6ca47bb85865f7207eca74 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.33.7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.label-schema.build-date=20260223) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:24 trial183 podman[75671]: 2026-02-24 16:51:24.123134008 +0000 UTC m=+0.053296596 container attach 69c1b82aab75577719e29424b4263597cf6295d26a6ca47bb85865f7207eca74 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-2-deactivate, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.build-date=20260223, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, io.buildah.version=1.33.7, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.schema-version=1.0, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-24T16:51:24.234 INFO:journalctl@ceph.osd.2.trial183.stdout:Feb 24 16:51:24 trial183 podman[75671]: 2026-02-24 16:51:24.079088895 +0000 UTC m=+0.009251473 image pull c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18 2026-02-24T16:51:24.287 DEBUG:teuthology.orchestra.run.trial183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.2.service' 2026-02-24T16:51:24.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:24.320 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-24T16:51:24.320 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2026-02-24T16:51:24.320 DEBUG:teuthology.orchestra.run.trial191:> sudo systemctl stop ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.3 2026-02-24T16:51:24.591 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:24 trial191 systemd[1]: Stopping Ceph osd.3 for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:24.591 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:51:24.431+0000 7ff9d09d4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-24T16:51:24.591 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:51:24.431+0000 7ff9d09d4640 -1 osd.3 32 *** Got signal Terminated *** 2026-02-24T16:51:24.591 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:24 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3[47620]: 2026-02-24T16:51:24.431+0000 7ff9d09d4640 -1 osd.3 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74531]: 2026-02-24 16:51:29.652524781 +0000 UTC m=+5.232277203 container died 857ad55401df9195a08749b80099f77d57016a2879e1096b75d9c233280166f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260223, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.33.7, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.license=GPLv2, CEPH_REF=wip-yuri8-testing-2026-02-23-2128) 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74531]: 2026-02-24 16:51:29.661535308 +0000 UTC m=+5.241287730 container remove 857ad55401df9195a08749b80099f77d57016a2879e1096b75d9c233280166f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260223, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.33.7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 bash[74531]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.794466685 +0000 UTC m=+0.014770362 container create a1aed52aa11f77acd91f170fe8a473830e6046791ef8b97b4121388b4b53cf9a (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3-deactivate, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260223, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.818550833 +0000 UTC m=+0.038854510 container init a1aed52aa11f77acd91f170fe8a473830e6046791ef8b97b4121388b4b53cf9a (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260223, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True) 2026-02-24T16:51:29.956 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.820911365 +0000 UTC m=+0.041215052 container start a1aed52aa11f77acd91f170fe8a473830e6046791ef8b97b4121388b4b53cf9a (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, ceph=True, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, io.buildah.version=1.33.7, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260223) 2026-02-24T16:51:29.957 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.82146273 +0000 UTC m=+0.041766407 container attach a1aed52aa11f77acd91f170fe8a473830e6046791ef8b97b4121388b4b53cf9a (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3-deactivate, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, io.buildah.version=1.33.7, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-24T16:51:29.957 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.788330845 +0000 UTC m=+0.008634522 image pull c2b2dcd214a8767139a8bc7faca7ab40a17d4be3761d6db8e2295a754d77ebf7 quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18 2026-02-24T16:51:29.957 INFO:journalctl@ceph.osd.3.trial191.stdout:Feb 24 16:51:29 trial191 podman[74673]: 2026-02-24 16:51:29.965164448 +0000 UTC m=+0.185468135 container died a1aed52aa11f77acd91f170fe8a473830e6046791ef8b97b4121388b4b53cf9a (image=quay.ceph.io/ceph-ci/ceph@sha256:0578ae50e4384f06da702540f3743781410d0c1737da0801db05e26214110b18, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-osd-3-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-yuri8-testing-2026-02-23-2128, org.label-schema.build-date=20260223, CEPH_SHA1=618c318a7a0b287ab5aeff207b7bf1d73ef45044, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.7) 2026-02-24T16:51:29.984 DEBUG:teuthology.orchestra.run.trial191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6deba16a-119f-11f1-a847-d404e6e7d460@osd.3.service' 2026-02-24T16:51:30.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-24T16:51:30.020 INFO:tasks.cephadm.osd.3:Stopped osd.3 2026-02-24T16:51:30.020 DEBUG:teuthology.orchestra.run.trial051:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:30.139 INFO:teuthology.orchestra.run.trial051.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:30.372 DEBUG:teuthology.orchestra.run.trial052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:30.490 INFO:teuthology.orchestra.run.trial052.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:30.734 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:30.851 INFO:teuthology.orchestra.run.trial151.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:31.667 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:31.785 INFO:teuthology.orchestra.run.trial153.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:32.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:32 trial153 systemd[1]: Stopping Ceph nvmeof.nvmeof.b for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:32.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:51:32] INFO server.py:42 (2): GatewayServer: SIGTERM received signum=15 2026-02-24T16:51:32.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:51:32] INFO server.py:197 (2): GatewayServer is terminating gracefully... 2026-02-24T16:51:32.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:51:32] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.b) pid 19 args ['/usr/bin/ceph-nvmeof-monitor-client', '--gateway-name', 'client.nvmeof.nvmeof.b', '--gateway-address', '10.20.193.153:5500', '--gateway-pool', 'mypool', '--gateway-group', 'mygroup0', '--monitor-group-address', '10.20.193.153:5499', '-c', '/etc/ceph/ceph.conf', '-n', 'client.nvmeof.nvmeof.b', '-k', '/etc/ceph/keyring'] ... 2026-02-24T16:51:32.314 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:32 trial153 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b[52972]: [24-Feb-2026 16:51:32] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.b) pid 127 args ['/usr/local/bin/nvmf_tgt', '--wait-for-rpc', '-u', '-r', '/var/tmp/spdk.sock', '--lcores', '(0-3)'] ... 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 bash[72311]: time="2026-02-24T16:51:42Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b in 10 seconds, resorting to SIGKILL" 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 podman[72311]: 2026-02-24 16:51:42.207804257 +0000 UTC m=+10.032273502 container died 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b, vcs-type=git, io.ceph.nvmeof=, name=ceph-nvmeof, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, release=, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vendor=Red Hat, Inc., vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, summary=Ceph NVMe over Fabrics Gateway, org.centos.version=9.0-30.el9, io.spdk.build.ceph-release=20.2.0, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.openshift.wants=ceph,rbd, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, distribution-scope=public, io.spdk.build.target-arch=x86-64-v2, maintainer=Ceph Developers , io.spdk.build.pkgdep-args=--rbd, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.makeflags=, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, version=1.7.1, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.maintainer=Ceph Developers , io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.spdk.git.branch=undefined, architecture=x86_64, io.buildah.version=1.29.0, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=25.09, build-date=2026-02-24 13:05:47 UTC, url=https://github.com/ceph/ceph-nvmeof, io.ceph.component=ceph-nvmeof, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.branch=devel, io.ceph.version=1.7.1, com.redhat.component=ubi9-container, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03) 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 podman[72311]: 2026-02-24 16:51:42.219043312 +0000 UTC m=+10.043512558 container remove 9c820102d84c32077628e1d3182e1992b12bcbf3896a666bc2ad8e2a67529daf (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b, version=1.7.1, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.version=25.09, io.ceph.nvmeof=, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, url=https://github.com/ceph/ceph-nvmeof, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.git.branch=undefined, maintainer=Ceph Developers , vendor=Red Hat, Inc., build-date=2026-02-24 13:05:47 UTC, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.build.makeflags=, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., name=ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.openshift.wants=ceph,rbd, io.ceph.component=ceph-nvmeof, io.spdk.build.target-arch=x86-64-v2, org.centos.version=9.0-30.el9, vcs-type=git, architecture=x86_64, io.spdk.git.repo=https://github.com/ceph/spdk.git, release=, distribution-scope=public, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.component=ubi9-container, summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.branch=devel, io.ceph.maintainer=Ceph Developers , io.spdk.build.pkgdep-args=--rbd, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.version=1.7.1, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.ceph-release=20.2.0) 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 bash[72311]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-b 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.b.service: Main process exited, code=exited, status=137/n/a 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.b.service: Failed with result 'exit-code'. 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 systemd[1]: Stopped Ceph nvmeof.nvmeof.b for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:51:42.518 INFO:journalctl@ceph.nvmeof.nvmeof.b.trial153.stdout:Feb 24 16:51:42 trial153 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.b.service: Consumed 35min 55.527s CPU time. 2026-02-24T16:51:42.805 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:42.922 INFO:teuthology.orchestra.run.trial183.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:43.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:43 trial183 systemd[1]: Stopping Ceph nvmeof.nvmeof.c for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:43.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:51:43] INFO server.py:42 (2): GatewayServer: SIGTERM received signum=15 2026-02-24T16:51:43.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:51:43] INFO server.py:197 (2): GatewayServer is terminating gracefully... 2026-02-24T16:51:43.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:51:43] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.c) pid 19 args ['/usr/bin/ceph-nvmeof-monitor-client', '--gateway-name', 'client.nvmeof.nvmeof.c', '--gateway-address', '10.20.193.183:5500', '--gateway-pool', 'mypool', '--gateway-group', 'mygroup0', '--monitor-group-address', '10.20.193.183:5499', '-c', '/etc/ceph/ceph.conf', '-n', 'client.nvmeof.nvmeof.c', '-k', '/etc/ceph/keyring'] ... 2026-02-24T16:51:43.484 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:43 trial183 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c[56637]: [24-Feb-2026 16:51:43] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.c) pid 100 args ['/usr/local/bin/nvmf_tgt', '--wait-for-rpc', '-u', '-r', '/var/tmp/spdk.sock', '--lcores', '(0-3)'] ... 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 bash[75974]: time="2026-02-24T16:51:53Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c in 10 seconds, resorting to SIGKILL" 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 podman[75974]: 2026-02-24 16:51:53.315618433 +0000 UTC m=+10.032490986 container died d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c, io.ceph.component=ceph-nvmeof, release=, vendor=Red Hat, Inc., vcs-type=git, io.spdk.git.branch=undefined, io.ceph.git.branch=devel, url=https://github.com/ceph/ceph-nvmeof, build-date=2026-02-24 13:05:47 UTC, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.ceph-release=20.2.0, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.spdk.build.pkgdep-args=--rbd, io.spdk.build.makeflags=, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.version=1.7.1, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.openshift.wants=ceph,rbd, io.ceph.nvmeof=, architecture=x86_64, io.buildah.version=1.29.0, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, name=ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.maintainer=Ceph Developers , io.spdk.git.repo=https://github.com/ceph/spdk.git, version=1.7.1, io.ceph.summary=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , io.spdk.version=25.09, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, com.redhat.component=ubi9-container, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.url=https://github.com/ceph/ceph-nvmeof, distribution-scope=public, org.centos.version=9.0-30.el9, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 podman[75974]: 2026-02-24 16:51:53.327067655 +0000 UTC m=+10.043940208 container remove d325550ecdd174a4dd910ff17cf9e965ca538166978ba6c87b0b4c3240f540c5 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=devel, vendor=Red Hat, Inc., io.spdk.version=25.09, io.spdk.build.makeflags=, io.spdk.git.branch=undefined, build-date=2026-02-24 13:05:47 UTC, io.ceph.component=ceph-nvmeof, release=, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.openshift.wants=ceph,rbd, io.ceph.nvmeof=, io.spdk.build.ceph-release=20.2.0, vcs-type=git, version=1.7.1, io.spdk.git.repo=https://github.com/ceph/spdk.git, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, name=ceph-nvmeof, io.ceph.maintainer=Ceph Developers , org.centos.version=9.0-30.el9, com.redhat.component=ubi9-container, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.buildah.version=1.29.0, summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=x86-64-v2, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.version=1.7.1, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, architecture=x86_64, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.pkgdep-args=--rbd, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom) 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 bash[75974]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-c 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.c.service: Main process exited, code=exited, status=137/n/a 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.c.service: Failed with result 'exit-code'. 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 systemd[1]: Stopped Ceph nvmeof.nvmeof.c for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:51:53.648 INFO:journalctl@ceph.nvmeof.nvmeof.c.trial183.stdout:Feb 24 16:51:53 trial183 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.c.service: Consumed 36min 38.406s CPU time. 2026-02-24T16:51:53.909 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force --keep-logs 2026-02-24T16:51:54.028 INFO:teuthology.orchestra.run.trial191.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:51:54.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:51:54 trial191 systemd[1]: Stopping Ceph nvmeof.nvmeof.d for 6deba16a-119f-11f1-a847-d404e6e7d460... 2026-02-24T16:51:54.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:51:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:51:54] INFO server.py:42 (2): GatewayServer: SIGTERM received signum=15 2026-02-24T16:51:54.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:51:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:51:54] INFO server.py:197 (2): GatewayServer is terminating gracefully... 2026-02-24T16:51:54.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:51:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:51:54] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.d) pid 19 args ['/usr/bin/ceph-nvmeof-monitor-client', '--gateway-name', 'client.nvmeof.nvmeof.d', '--gateway-address', '10.20.193.191:5500', '--gateway-pool', 'mypool', '--gateway-group', 'mygroup0', '--monitor-group-address', '10.20.193.191:5499', '-c', '/etc/ceph/ceph.conf', '-n', 'client.nvmeof.nvmeof.d', '-k', '/etc/ceph/keyring'] ... 2026-02-24T16:51:54.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:51:54 trial191 ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d[56033]: [24-Feb-2026 16:51:54] INFO server.py:739 (2): Terminating sub process of (client.nvmeof.nvmeof.d) pid 121 args ['/usr/local/bin/nvmf_tgt', '--wait-for-rpc', '-u', '-r', '/var/tmp/spdk.sock', '--lcores', '(0-3)'] ... 2026-02-24T16:52:04.591 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 bash[74947]: time="2026-02-24T16:52:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d in 10 seconds, resorting to SIGKILL" 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 podman[74947]: 2026-02-24 16:52:04.301546165 +0000 UTC m=+10.033759880 container died 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.build.target-arch=x86-64-v2, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=devel, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.repo=https://github.com/ceph/spdk.git, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2026-02-24 13:05:47 UTC, io.ceph.nvmeof=, io.spdk.build.pkgdep-args=--rbd, io.spdk.version=25.09, distribution-scope=public, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.ceph-release=20.2.0, maintainer=Ceph Developers , io.ceph.version=1.7.1, io.ceph.summary=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.wants=ceph,rbd, io.spdk.build.makeflags=, org.centos.version=9.0-30.el9, com.redhat.component=ubi9-container, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, release=, version=1.7.1, vendor=Red Hat, Inc., io.ceph.maintainer=Ceph Developers , name=ceph-nvmeof, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, io.buildah.version=1.29.0, url=https://github.com/ceph/ceph-nvmeof, io.ceph.component=ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, architecture=x86_64, io.spdk.git.branch=undefined, summary=Ceph NVMe over Fabrics Gateway) 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 podman[74947]: 2026-02-24 16:52:04.314423694 +0000 UTC m=+10.046637409 container remove 9d7ade3ab44ff6a069d3e56c1459efb5207aa535529277f4f22a47442e57dbb8 (image=quay.io/ceph/nvmeof:devel, name=ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d, io.spdk.build.target-arch=x86-64-v2, io.spdk.build.configure-args=--with-idxd --with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, org.centos.version=9.0-30.el9, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.pkgdep-args=--rbd, io.spdk.git.branch=undefined, maintainer=Ceph Developers , io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.version=1.7.1, io.spdk.build.makeflags=, io.ceph.nvmeof=, vcs-type=git, io.ceph.maintainer=Ceph Developers , io.spdk.build.ceph-release=20.2.0, version=1.7.1, summary=Ceph NVMe over Fabrics Gateway, vcs-ref=5b3adc3ce6aa283baaa0c72d4265014805a14f03, distribution-scope=public, io.ceph.component=ceph-nvmeof, release=, io.buildah.version=1.29.0, io.spdk.git.repo=https://github.com/ceph/spdk.git, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2026-02-24 13:05:47 UTC, io.spdk.version=25.09, com.redhat.component=ubi9-container, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.branch=devel, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.git.commit=5b3adc3ce6aa283baaa0c72d4265014805a14f03, io.openshift.wants=ceph,rbd, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, name=ceph-nvmeof, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.git.commit=cf5921356df981b1954fa7e2fefecb1c3242c8f9, url=https://github.com/ceph/ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage) 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 bash[74947]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460-nvmeof-nvmeof-d 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.d.service: Main process exited, code=exited, status=137/n/a 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.d.service: Failed with result 'exit-code'. 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 systemd[1]: Stopped Ceph nvmeof.nvmeof.d for 6deba16a-119f-11f1-a847-d404e6e7d460. 2026-02-24T16:52:04.592 INFO:journalctl@ceph.nvmeof.nvmeof.d.trial191.stdout:Feb 24 16:52:04 trial191 systemd[1]: ceph-6deba16a-119f-11f1-a847-d404e6e7d460@nvmeof.nvmeof.d.service: Consumed 37min 14.001s CPU time. 2026-02-24T16:52:04.865 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:04.894 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:04.922 DEBUG:teuthology.orchestra.run.trial151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:04.950 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:04.979 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:05.010 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2026-02-24T16:52:05.039 INFO:tasks.cephadm:Archiving crash dumps... 2026-02-24T16:52:05.042 DEBUG:teuthology.misc:Transferring archived files from trial051:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial051/crash 2026-02-24T16:52:05.043 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.070 INFO:teuthology.orchestra.run.trial051.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.070 INFO:teuthology.orchestra.run.trial051.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.072 DEBUG:teuthology.misc:Transferring archived files from trial052:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial052/crash 2026-02-24T16:52:05.073 DEBUG:teuthology.orchestra.run.trial052:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.100 INFO:teuthology.orchestra.run.trial052.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.100 INFO:teuthology.orchestra.run.trial052.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.103 DEBUG:teuthology.misc:Transferring archived files from trial151:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial151/crash 2026-02-24T16:52:05.105 DEBUG:teuthology.orchestra.run.trial151:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.132 INFO:teuthology.orchestra.run.trial151.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.132 INFO:teuthology.orchestra.run.trial151.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.134 DEBUG:teuthology.misc:Transferring archived files from trial153:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial153/crash 2026-02-24T16:52:05.135 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.161 INFO:teuthology.orchestra.run.trial153.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.161 INFO:teuthology.orchestra.run.trial153.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.164 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial183/crash 2026-02-24T16:52:05.164 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.191 INFO:teuthology.orchestra.run.trial183.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.191 INFO:teuthology.orchestra.run.trial183.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.193 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial191/crash 2026-02-24T16:52:05.194 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash -- . 2026-02-24T16:52:05.221 INFO:teuthology.orchestra.run.trial191.stderr:tar: /var/lib/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/crash: Cannot open: No such file or directory 2026-02-24T16:52:05.221 INFO:teuthology.orchestra.run.trial191.stderr:tar: Error is not recoverable: exiting now 2026-02-24T16:52:05.222 INFO:tasks.cephadm:Checking cluster log for badness... 2026-02-24T16:52:05.223 DEBUG:teuthology.orchestra.run.trial151:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v OSD_DOWN | grep -E -v OSD_HOST_DOWN | grep -E -v OSD_ROOT_DOWN | grep -E -v PG_DEGRADED | grep -E -v NVMEOF_SINGLE_GATEWAY | grep -E -v CEPHADM_FAILED_DAEMON | head -n 1 2026-02-24T16:52:05.249 INFO:tasks.cephadm:Compressing logs... 2026-02-24T16:52:05.249 DEBUG:teuthology.orchestra.run.trial051:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.251 DEBUG:teuthology.orchestra.run.trial052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.253 DEBUG:teuthology.orchestra.run.trial151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.274 INFO:teuthology.orchestra.run.trial051.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory 2026-02-24T16:52:05.274 INFO:teuthology.orchestra.run.trial051.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2026-02-24T16:52:05.275 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.275 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/cephadm.log: 84.4% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.275 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: 87.1% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.276 INFO:teuthology.orchestra.run.trial052.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2026-02-24T16:52:05.276 INFO:teuthology.orchestra.run.trial052.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2026-02-24T16:52:05.276 INFO:teuthology.orchestra.run.trial052.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.276 INFO:teuthology.orchestra.run.trial052.stderr:/var/log/ceph/cephadm.log: 84.2% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.277 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-24T16:52:05.277 INFO:teuthology.orchestra.run.trial051.stderr:real 0m0.011s 2026-02-24T16:52:05.277 INFO:teuthology.orchestra.run.trial051.stderr:user 0m0.006s 2026-02-24T16:52:05.277 INFO:teuthology.orchestra.run.trial051.stderr:sys 0m0.014s 2026-02-24T16:52:05.277 INFO:teuthology.orchestra.run.trial052.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: 87.1% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.278 INFO:teuthology.orchestra.run.trial052.stderr: 2026-02-24T16:52:05.278 INFO:teuthology.orchestra.run.trial052.stderr:real 0m0.011s 2026-02-24T16:52:05.278 INFO:teuthology.orchestra.run.trial052.stderr:user 0m0.006s 2026-02-24T16:52:05.278 INFO:teuthology.orchestra.run.trial052.stderr:sys 0m0.014s 2026-02-24T16:52:05.291 DEBUG:teuthology.orchestra.run.trial153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.293 DEBUG:teuthology.orchestra.run.trial183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.295 DEBUG:teuthology.orchestra.run.trial191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:52:05.313 INFO:teuthology.orchestra.run.trial151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2026-02-24T16:52:05.313 INFO:teuthology.orchestra.run.trial151.stderr:: No such file or directory 2026-02-24T16:52:05.313 INFO:teuthology.orchestra.run.trial151.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.a.log 2026-02-24T16:52:05.313 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.0.log 2026-02-24T16:52:05.313 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log 2026-02-24T16:52:05.314 INFO:teuthology.orchestra.run.trial151.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log 2026-02-24T16:52:05.314 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.0.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.314 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mgr.x.log 2026-02-24T16:52:05.314 INFO:teuthology.orchestra.run.trial151.stderr: 83.5% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log.gz 2026-02-24T16:52:05.314 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log 2026-02-24T16:52:05.315 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial151.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log: 90.3% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log.gz 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial153.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial151.stderr: 87.5% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log.gz 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial151.stderr: 93.8% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2026-02-24T16:52:05.316 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log 2026-02-24T16:52:05.317 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log 2026-02-24T16:52:05.317 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.b.log 2026-02-24T16:52:05.317 INFO:teuthology.orchestra.run.trial153.stderr: 83.3% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log.gz 2026-02-24T16:52:05.317 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose --/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.317 INFO:teuthology.orchestra.run.trial153.stderr: 86.4% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.318 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.1.log 2026-02-24T16:52:05.318 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log 2026-02-24T16:52:05.318 INFO:teuthology.orchestra.run.trial183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2026-02-24T16:52:05.318 INFO:teuthology.orchestra.run.trial183.stderr:: No such file or directory 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.1.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log: 90.4% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log.gz 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial153.stderr: 93.2% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.2.log 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial153.stderr: 87.6% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log.gz 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.c.log 2026-02-24T16:52:05.319 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.2.log: 87.6% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.3.log 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.c.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log: 83.3% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.cephadm.log.gz 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log 2026-02-24T16:52:05.320 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log 2026-02-24T16:52:05.321 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.3.log: 86.9% -- replaced with /var/log/ceph/cephadm.log.gz 2026-02-24T16:52:05.321 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log: 87.6% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.log.gz 2026-02-24T16:52:05.321 INFO:teuthology.orchestra.run.trial183.stderr: 90.4% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph.audit.log.gz 2026-02-24T16:52:05.322 INFO:teuthology.orchestra.run.trial183.stderr: 93.9% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.323 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log: 93.8% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-volume.log.gz 2026-02-24T16:52:05.446 INFO:teuthology.orchestra.run.trial151.stderr: 91.9% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mgr.x.log.gz 2026-02-24T16:52:05.543 INFO:teuthology.orchestra.run.trial153.stderr: 93.6% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.b.log.gz 2026-02-24T16:52:05.549 INFO:teuthology.orchestra.run.trial183.stderr: 93.5% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.c.log.gz 2026-02-24T16:52:05.801 INFO:teuthology.orchestra.run.trial151.stderr: 92.2% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-mon.a.log.gz 2026-02-24T16:52:47.539 INFO:teuthology.orchestra.run.trial153.stderr: 92.6% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.1.log.gz 2026-02-24T16:52:47.541 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-24T16:52:47.541 INFO:teuthology.orchestra.run.trial153.stderr:real 0m42.233s 2026-02-24T16:52:47.541 INFO:teuthology.orchestra.run.trial153.stderr:user 0m40.754s 2026-02-24T16:52:47.541 INFO:teuthology.orchestra.run.trial153.stderr:sys 0m1.619s 2026-02-24T16:52:48.372 INFO:teuthology.orchestra.run.trial151.stderr: 92.5% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.0.log.gz 2026-02-24T16:52:48.373 INFO:teuthology.orchestra.run.trial151.stderr: 2026-02-24T16:52:48.373 INFO:teuthology.orchestra.run.trial151.stderr:real 0m43.069s 2026-02-24T16:52:48.373 INFO:teuthology.orchestra.run.trial151.stderr:user 0m41.991s 2026-02-24T16:52:48.373 INFO:teuthology.orchestra.run.trial151.stderr:sys 0m1.611s 2026-02-24T16:52:53.431 INFO:teuthology.orchestra.run.trial191.stderr: 92.6% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.3.log.gz 2026-02-24T16:52:53.434 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-24T16:52:53.434 INFO:teuthology.orchestra.run.trial191.stderr:real 0m48.121s 2026-02-24T16:52:53.434 INFO:teuthology.orchestra.run.trial191.stderr:user 0m46.217s 2026-02-24T16:52:53.434 INFO:teuthology.orchestra.run.trial191.stderr:sys 0m1.804s 2026-02-24T16:52:57.469 INFO:teuthology.orchestra.run.trial183.stderr: 92.5% -- replaced with /var/log/ceph/6deba16a-119f-11f1-a847-d404e6e7d460/ceph-osd.2.log.gz 2026-02-24T16:52:57.471 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-24T16:52:57.471 INFO:teuthology.orchestra.run.trial183.stderr:real 0m52.160s 2026-02-24T16:52:57.471 INFO:teuthology.orchestra.run.trial183.stderr:user 0m50.258s 2026-02-24T16:52:57.471 INFO:teuthology.orchestra.run.trial183.stderr:sys 0m2.021s 2026-02-24T16:52:57.471 INFO:tasks.cephadm:Archiving logs... 2026-02-24T16:52:57.471 DEBUG:teuthology.misc:Transferring archived files from trial051:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial051/log 2026-02-24T16:52:57.473 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:52:57.502 DEBUG:teuthology.misc:Transferring archived files from trial052:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial052/log 2026-02-24T16:52:57.504 DEBUG:teuthology.orchestra.run.trial052:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:52:57.534 DEBUG:teuthology.misc:Transferring archived files from trial151:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial151/log 2026-02-24T16:52:57.535 DEBUG:teuthology.orchestra.run.trial151:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:53:01.137 DEBUG:teuthology.misc:Transferring archived files from trial153:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial153/log 2026-02-24T16:53:01.139 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:53:04.581 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial183/log 2026-02-24T16:53:04.582 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:53:08.916 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial191/log 2026-02-24T16:53:08.917 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:53:12.772 INFO:tasks.cephadm:Removing cluster... 2026-02-24T16:53:12.773 DEBUG:teuthology.orchestra.run.trial051:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:12.891 INFO:teuthology.orchestra.run.trial051.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:13.126 DEBUG:teuthology.orchestra.run.trial052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:13.244 INFO:teuthology.orchestra.run.trial052.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:13.482 DEBUG:teuthology.orchestra.run.trial151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:13.600 INFO:teuthology.orchestra.run.trial151.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:13.853 DEBUG:teuthology.orchestra.run.trial153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:13.971 INFO:teuthology.orchestra.run.trial153.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:14.209 DEBUG:teuthology.orchestra.run.trial183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:14.326 INFO:teuthology.orchestra.run.trial183.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:14.582 DEBUG:teuthology.orchestra.run.trial191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6deba16a-119f-11f1-a847-d404e6e7d460 --force 2026-02-24T16:53:14.700 INFO:teuthology.orchestra.run.trial191.stdout:Deleting cluster with fsid: 6deba16a-119f-11f1-a847-d404e6e7d460 2026-02-24T16:53:14.954 INFO:tasks.cephadm:Removing cephadm ... 2026-02-24T16:53:14.954 DEBUG:teuthology.orchestra.run.trial051:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:14.974 DEBUG:teuthology.orchestra.run.trial052:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:14.994 DEBUG:teuthology.orchestra.run.trial151:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:15.013 DEBUG:teuthology.orchestra.run.trial153:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:15.032 DEBUG:teuthology.orchestra.run.trial183:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:15.051 DEBUG:teuthology.orchestra.run.trial191:> rm -rf /home/ubuntu/cephtest/cephadm 2026-02-24T16:53:15.070 INFO:tasks.cephadm:Teardown complete 2026-02-24T16:53:15.070 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:53:15.087 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T16:53:15.087 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.090 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.093 DEBUG:teuthology.orchestra.run.trial151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.096 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.099 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.101 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:53:15.543 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T16:53:15.555 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T16:53:15.555 DEBUG:teuthology.orchestra.run.trial051:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.557 DEBUG:teuthology.orchestra.run.trial052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.558 DEBUG:teuthology.orchestra.run.trial151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.560 DEBUG:teuthology.orchestra.run.trial153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.562 DEBUG:teuthology.orchestra.run.trial183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.563 DEBUG:teuthology.orchestra.run.trial191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:53:15.571 INFO:teuthology.orchestra.run.trial051.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.574 INFO:teuthology.orchestra.run.trial052.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.574 INFO:teuthology.orchestra.run.trial151.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.576 INFO:teuthology.orchestra.run.trial153.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.578 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.580 INFO:teuthology.orchestra.run.trial191.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:=============================================================================== 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:^+ h134-215-155-177.mdtnwi.> 2 6 377 27 +7814us[+7814us] +/- 22ms 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 28 -29ms[ -29ms] +/- 42ms 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:^- time1.lshiy.com 2 6 267 30 +5822us[+5822us] +/- 117ms 2026-02-24T16:53:15.597 INFO:teuthology.orchestra.run.trial051.stdout:^* 144.202.62.209.vultruser> 3 6 377 31 +5855us[+5895us] +/- 19ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:=============================================================================== 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:^+ h134-215-155-177.mdtnwi.> 2 6 377 29 +7572us[+7590us] +/- 22ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 29 -29ms[ -29ms] +/- 42ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:^- time1.lshiy.com 2 7 377 28 +5991us[+5991us] +/- 118ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial151.stdout:^* 144.202.62.209.vultruser> 3 6 377 28 +6044us[+6063us] +/- 19ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:=============================================================================== 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:^+ h134-215-155-177.mdtnwi.> 2 6 377 29 +7923us[+7923us] +/- 22ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 26 -29ms[ -29ms] +/- 42ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:^- time1.lshiy.com 2 7 367 30 +6743us[+6743us] +/- 122ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial052.stdout:^* 144.202.62.209.vultruser> 3 6 377 30 +6136us[+6186us] +/- 19ms 2026-02-24T16:53:15.598 INFO:teuthology.orchestra.run.trial153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial153.stdout:=============================================================================== 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial153.stdout:^+ h134-215-155-177.mdtnwi.> 2 6 377 28 +6730us[+6730us] +/- 22ms 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial153.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 27 -29ms[ -29ms] +/- 42ms 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial153.stdout:^- time1.lshiy.com 2 6 367 30 +6391us[+6325us] +/- 121ms 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial153.stdout:^* 144.202.62.209.vultruser> 3 6 377 28 +5956us[+5890us] +/- 19ms 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.599 INFO:teuthology.orchestra.run.trial191.stdout:=============================================================================== 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial191.stdout:^- 50.218.103.254 2 6 377 98 +12ms[ +12ms] +/- 29ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial191.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 31 -24ms[ -24ms] +/- 42ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial191.stdout:^- time1.lshiy.com 2 6 377 287 +11ms[ +11ms] +/- 124ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial191.stdout:^* 144.202.62.209.vultruser> 3 6 377 30 +11ms[ +11ms] +/- 19ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:^+ h134-215-155-177.mdtnwi.> 2 6 377 28 +7660us[+7660us] +/- 21ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:^+ t2.time.bf1.yahoo.com 2 6 377 28 -29ms[ -29ms] +/- 42ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:^- time1.lshiy.com 2 6 377 28 +6604us[+6604us] +/- 117ms 2026-02-24T16:53:15.600 INFO:teuthology.orchestra.run.trial183.stdout:^* 144.202.62.209.vultruser> 3 6 377 29 +6243us[+6242us] +/- 19ms 2026-02-24T16:53:15.600 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T16:53:15.611 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T16:53:15.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T16:53:15.624 DEBUG:teuthology.orchestra.run.trial051:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.705 DEBUG:teuthology.orchestra.run.trial052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.765 DEBUG:teuthology.orchestra.run.trial151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.824 DEBUG:teuthology.orchestra.run.trial153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.882 DEBUG:teuthology.orchestra.run.trial183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.938 DEBUG:teuthology.orchestra.run.trial191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:53:15.994 DEBUG:teuthology.orchestra.run.trial051:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.025 DEBUG:teuthology.orchestra.run.trial052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.055 DEBUG:teuthology.orchestra.run.trial151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.085 DEBUG:teuthology.orchestra.run.trial153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.114 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.142 DEBUG:teuthology.orchestra.run.trial191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:53:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:16.171 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T16:53:16.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T16:53:16.205 INFO:teuthology.task.internal:Duration was 1172.657112 seconds 2026-02-24T16:53:16.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T16:53:16.222 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T16:53:16.223 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.226 DEBUG:teuthology.orchestra.run.trial052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.229 DEBUG:teuthology.orchestra.run.trial151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.231 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.234 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.237 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:53:16.262 INFO:teuthology.orchestra.run.trial051.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.264 INFO:teuthology.orchestra.run.trial052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.268 INFO:teuthology.orchestra.run.trial151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.270 INFO:teuthology.orchestra.run.trial153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.271 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.273 INFO:teuthology.orchestra.run.trial191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:53:16.767 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T16:53:16.767 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial051.front.sepia.ceph.com 2026-02-24T16:53:16.768 DEBUG:teuthology.orchestra.run.trial051:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.788 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial052.front.sepia.ceph.com 2026-02-24T16:53:16.789 DEBUG:teuthology.orchestra.run.trial052:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.809 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial151.front.sepia.ceph.com 2026-02-24T16:53:16.810 DEBUG:teuthology.orchestra.run.trial151:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.831 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial153.front.sepia.ceph.com 2026-02-24T16:53:16.831 DEBUG:teuthology.orchestra.run.trial153:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.851 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial183.front.sepia.ceph.com 2026-02-24T16:53:16.851 DEBUG:teuthology.orchestra.run.trial183:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial191.front.sepia.ceph.com 2026-02-24T16:53:16.870 DEBUG:teuthology.orchestra.run.trial191:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:53:16.890 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T16:53:16.891 DEBUG:teuthology.orchestra.run.trial051:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:16.893 DEBUG:teuthology.orchestra.run.trial052:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:16.896 DEBUG:teuthology.orchestra.run.trial151:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:16.899 DEBUG:teuthology.orchestra.run.trial153:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:16.901 DEBUG:teuthology.orchestra.run.trial183:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:16.913 DEBUG:teuthology.orchestra.run.trial191:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.026 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T16:53:17.026 DEBUG:teuthology.orchestra.run.trial051:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.028 DEBUG:teuthology.orchestra.run.trial052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.030 DEBUG:teuthology.orchestra.run.trial151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.031 DEBUG:teuthology.orchestra.run.trial153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.033 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.035 DEBUG:teuthology.orchestra.run.trial191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:53:17.049 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.050 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.050 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.050 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial052.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial052.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial052.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial052.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.051 INFO:teuthology.orchestra.run.trial052.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.052 INFO:teuthology.orchestra.run.trial151.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.053 INFO:teuthology.orchestra.run.trial151.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.053 INFO:teuthology.orchestra.run.trial151.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.053 INFO:teuthology.orchestra.run.trial151.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.053 INFO:teuthology.orchestra.run.trial151.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.056 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.057 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.057 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.057 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.058 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial191.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:53:17.060 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:53:17.061 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:53:17.061 INFO:teuthology.orchestra.run.trial153.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:53:17.061 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:53:17.066 INFO:teuthology.orchestra.run.trial052.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 86.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.067 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 87.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.072 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 85.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.073 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 86.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.073 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 86.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.077 INFO:teuthology.orchestra.run.trial151.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:53:17.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T16:53:17.096 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T16:53:17.096 DEBUG:teuthology.orchestra.run.trial051:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.133 DEBUG:teuthology.orchestra.run.trial052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.157 DEBUG:teuthology.orchestra.run.trial151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.184 DEBUG:teuthology.orchestra.run.trial153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.209 DEBUG:teuthology.orchestra.run.trial183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.234 DEBUG:teuthology.orchestra.run.trial191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:53:17.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T16:53:17.278 DEBUG:teuthology.orchestra.run.trial051:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.281 DEBUG:teuthology.orchestra.run.trial052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.284 DEBUG:teuthology.orchestra.run.trial151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.287 DEBUG:teuthology.orchestra.run.trial153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.289 DEBUG:teuthology.orchestra.run.trial183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.292 DEBUG:teuthology.orchestra.run.trial191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.304 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.306 INFO:teuthology.orchestra.run.trial052.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.309 INFO:teuthology.orchestra.run.trial151.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.313 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.314 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.326 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern = core 2026-02-24T16:53:17.343 DEBUG:teuthology.orchestra.run.trial051:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.378 DEBUG:teuthology.orchestra.run.trial052:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.394 DEBUG:teuthology.orchestra.run.trial151:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.412 DEBUG:teuthology.orchestra.run.trial153:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.431 DEBUG:teuthology.orchestra.run.trial183:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.449 DEBUG:teuthology.orchestra.run.trial191:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:53:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:53:17.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T16:53:17.484 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T16:53:17.486 DEBUG:teuthology.misc:Transferring archived files from trial051:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial051 2026-02-24T16:53:17.486 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.533 DEBUG:teuthology.misc:Transferring archived files from trial052:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial052 2026-02-24T16:53:17.533 DEBUG:teuthology.orchestra.run.trial052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.572 DEBUG:teuthology.misc:Transferring archived files from trial151:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial151 2026-02-24T16:53:17.573 DEBUG:teuthology.orchestra.run.trial151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.615 DEBUG:teuthology.misc:Transferring archived files from trial153:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial153 2026-02-24T16:53:17.615 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.654 DEBUG:teuthology.misc:Transferring archived files from trial183:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial183 2026-02-24T16:53:17.654 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.693 DEBUG:teuthology.misc:Transferring archived files from trial191:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/remote/trial191 2026-02-24T16:53:17.693 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:53:17.732 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T16:53:17.733 DEBUG:teuthology.orchestra.run.trial051:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.736 DEBUG:teuthology.orchestra.run.trial052:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.739 DEBUG:teuthology.orchestra.run.trial151:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.741 DEBUG:teuthology.orchestra.run.trial153:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.744 DEBUG:teuthology.orchestra.run.trial183:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.747 DEBUG:teuthology.orchestra.run.trial191:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:53:17.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T16:53:17.800 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T16:53:17.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T16:53:17.818 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T16:53:17.818 DEBUG:teuthology.orchestra.run.trial051:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.821 DEBUG:teuthology.orchestra.run.trial052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.824 DEBUG:teuthology.orchestra.run.trial151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.827 DEBUG:teuthology.orchestra.run.trial153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.829 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.832 DEBUG:teuthology.orchestra.run.trial191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:53:17.836 INFO:teuthology.orchestra.run.trial051.stdout: 920835 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.839 INFO:teuthology.orchestra.run.trial052.stdout: 917525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.842 INFO:teuthology.orchestra.run.trial153.stdout: 917521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.844 INFO:teuthology.orchestra.run.trial151.stdout: 917521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.847 INFO:teuthology.orchestra.run.trial183.stdout: 917525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.848 INFO:teuthology.orchestra.run.trial191.stdout: 917521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:53 /home/ubuntu/cephtest 2026-02-24T16:53:17.850 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T16:53:17.867 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T16:53:17.905 INFO:teuthology.run:Summary data: description: nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_namespaces} duration: 1172.6571118831635 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-24T16:53:17.905 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:53:18.016 INFO:teuthology.run:pass